Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-06 Thread via GitHub
HonahX commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1797949558 @Fokko Thanks for the explanation! > Ability to write the JSON to the object store (that was the intent of this PR). I think we already support this:https://github.

Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-06 Thread via GitHub
HonahX commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1797920066 Hi @vrd83. It depends on which catalog you want to use to alter the table. For the RestCatalog, this is not a prerequisite. To enable altering the write order, we can impleme

Re: [I] Remove properties from `JdbcUtil` [iceberg]

2023-11-06 Thread via GitHub
nastra closed issue #8989: Remove properties from `JdbcUtil` URL: https://github.com/apache/iceberg/issues/8989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] De-dup props in JdbcUtil [iceberg]

2023-11-06 Thread via GitHub
nastra merged PR #8992: URL: https://github.com/apache/iceberg/pull/8992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-06 Thread via GitHub
vrd83 commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1797875405 Guys, is this a prerequisite for altering the [write-ordered-by ](https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--write-ordered-by) on a table? -- This is an auto

Re: [I] Flink write iceberg bug(org.apache.iceberg.exceptions.NotFoundException) [iceberg]

2023-11-06 Thread via GitHub
lirui-apache commented on issue #5846: URL: https://github.com/apache/iceberg/issues/5846#issuecomment-1797782929 Sure, I'll open a PR for it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1384340506 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -500,6 +511,9 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifi

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1384338591 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.a

Re: [I] Spark:CALL [rewrite_manifests] error Manifest is missing [iceberg]

2023-11-06 Thread via GitHub
372242283 commented on issue #4161: URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1797425869 Spark:3.3 Iceberg:13.0 Encountering the same problem I also have this problem. I use the iceberg table of hive Catalog, and the operation is rewrite_data_file-> rewrite

Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-06 Thread via GitHub
liurenjie1024 commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1384272688 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify dele

Re: [PR] Parquet: don't throw exception on row group filters when reading INT96 column [iceberg]

2023-11-06 Thread via GitHub
manuzhang commented on code in PR #8988: URL: https://github.com/apache/iceberg/pull/8988#discussion_r1384231086 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java: ## @@ -199,7 +201,7 @@ public Boolean lt(BoundReference ref, Literal lit)

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-06 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1384194329 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata metada

Re: [I] Spark: inconsistency in rewrite data and summary [iceberg]

2023-11-06 Thread via GitHub
github-actions[bot] commented on issue #7463: URL: https://github.com/apache/iceberg/issues/7463#issuecomment-1797064259 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-06 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1384188720 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata metada

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-06 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1384165960 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384147958 ## api/src/main/java/org/apache/iceberg/transforms/PartitionSpecVisitor.java: ## @@ -121,17 +121,13 @@ static R visit(Schema schema, PartitionField field, Partiti

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1382167692 ## api/src/main/java/org/apache/iceberg/transforms/Transforms.java: ## @@ -129,10 +131,14 @@ public static Transform year(Type type) { case DATE: r

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1382170673 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -205,27 +208,56 @@ public String toString() { } public static class TimestampType extends Pr

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384147958 ## api/src/main/java/org/apache/iceberg/transforms/PartitionSpecVisitor.java: ## @@ -121,17 +121,13 @@ static R visit(Schema schema, PartitionField field, Partiti

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384147958 ## api/src/main/java/org/apache/iceberg/transforms/PartitionSpecVisitor.java: ## @@ -121,17 +121,13 @@ static R visit(Schema schema, PartitionField field, Partiti

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384147958 ## api/src/main/java/org/apache/iceberg/transforms/PartitionSpecVisitor.java: ## @@ -121,17 +121,13 @@ static R visit(Schema schema, PartitionField field, Partiti

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1382166328 ## api/src/main/java/org/apache/iceberg/transforms/Months.java: ## @@ -55,14 +57,13 @@ public boolean satisfiesOrderOf(Transform other) { } if (other in

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1382165939 ## api/src/main/java/org/apache/iceberg/transforms/Days.java: ## @@ -55,14 +56,14 @@ public boolean satisfiesOrderOf(Transform other) { } if (other inst

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384122853 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -589,17 +603,17 @@ private static String sanitizeNumber(Number value, String type) {

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384147958 ## api/src/main/java/org/apache/iceberg/transforms/PartitionSpecVisitor.java: ## @@ -121,17 +121,13 @@ static R visit(Schema schema, PartitionField field, Partiti

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-06 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1384122853 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -589,17 +603,17 @@ private static String sanitizeNumber(Number value, String type) {

[PR] Build: Bump mkdocs-material from 9.4.7 to 9.4.8 [iceberg-python]

2023-11-06 Thread via GitHub
dependabot[bot] opened a new pull request, #131: URL: https://github.com/apache/iceberg-python/pull/131 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.4.7 to 9.4.8. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdo

[PR] [wip] de-dup props [iceberg]

2023-11-06 Thread via GitHub
thomaschow opened a new pull request, #8992: URL: https://github.com/apache/iceberg/pull/8992 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-06 Thread via GitHub
emkornfield commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1384024269 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify delete

Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-06 Thread via GitHub
emkornfield commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1384024269 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify delete

Re: [I] Adding new columns (mergeSchema) [iceberg]

2023-11-06 Thread via GitHub
FabricioZGalvani commented on issue #8908: URL: https://github.com/apache/iceberg/issues/8908#issuecomment-1796301185 Hello everyone, After several attempts, I managed to solve the mergeSchema issue I was facing. The solution was to apply the following configuration. I suspect that t

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-06 Thread via GitHub
waifairer commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1383886741 ## mkdocs/docs/configuration.md: ## @@ -195,6 +195,19 @@ catalog: table-name: iceberg ``` +If you prefer to pass the credentials explicitly to the client

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383715528 ## core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682641 ## api/src/main/java/org/apache/iceberg/Scan.java: ## @@ -77,6 +78,21 @@ public interface Scan> { */ ThisT includeColumnStats(); + /** + * Create a new scan f

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682326 ## core/src/main/java/org/apache/iceberg/GenericDataFile.java: ## @@ -66,23 +68,31 @@ class GenericDataFile extends BaseFile implements DataFile { * Copy constructor

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1383682090 ## api/src/main/java/org/apache/iceberg/Scan.java: ## @@ -77,6 +78,21 @@ public interface Scan> { */ ThisT includeColumnStats(); + /** + * Create a new scan f

[PR] Docs: Update site-docs/spark-quickstart.md [iceberg]

2023-11-06 Thread via GitHub
stavdav143 opened a new pull request, #8991: URL: https://github.com/apache/iceberg/pull/8991 Local volume with warehouse/notebooks to be mounted on Minio service instead of Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Kafka Connect: Initial project setup and event data structures [iceberg]

2023-11-06 Thread via GitHub
bryanck commented on code in PR #8701: URL: https://github.com/apache/iceberg/pull/8701#discussion_r1383589051 ## kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/AvroUtil.java: ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1383492951 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java: ## @@ -497,6 +500,67 @@ public void testReadStreamWithSnapshotTypeO

Re: [I] support partition spec update in pyiceberg [iceberg-python]

2023-11-06 Thread via GitHub
puchengy commented on issue #124: URL: https://github.com/apache/iceberg-python/issues/124#issuecomment-1795046924 @Fokko Hi, we are interested in Hive catalog and rest catalog in future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] docs(readme): Add feature roadmap and support to readme [iceberg-go]

2023-11-06 Thread via GitHub
nastra merged PR #32: URL: https://github.com/apache/iceberg-go/pull/32 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[I] Cannot decode dictionary of type INT96 when reading imported Spark parquet table [iceberg]

2023-11-06 Thread via GitHub
manuzhang opened a new issue, #8990: URL: https://github.com/apache/iceberg/issues/8990 ### Apache Iceberg version 1.2.1 ### Query engine Spark ### Please describe the bug 🐞 The following exception was thrown when reading an imported Spark parquet table wit

Re: [PR] docs(readme): Add feature roadmap and support to readme [iceberg-go]

2023-11-06 Thread via GitHub
zeroshade commented on PR #32: URL: https://github.com/apache/iceberg-go/pull/32#issuecomment-1794974142 CC @nastra @Fokko @rdblue @coded9 @bitsondatadev @wolfeidau -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] docs(readme): Add feature roadmap and support to readme [iceberg-go]

2023-11-06 Thread via GitHub
zeroshade opened a new pull request, #32: URL: https://github.com/apache/iceberg-go/pull/32 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1383413298 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.ap

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1383409990 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -500,6 +511,9 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifie

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1383411335 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.ap

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1383403517 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier originalT

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1383401740 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier originalT

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-06 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1383363198 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383362391 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,389 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383360341 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,389 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383356758 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383350384 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383346182 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383344359 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383343370 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383338484 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383336007 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -115,42 +126,13 @@ public void initialize(String inputName, Map properties) { @Overri

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-06 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1383335036 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383319743 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383329498 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383319743 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-06 Thread via GitHub
cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1383313970 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFiles

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383296718 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383296211 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383293089 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383292093 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383290717 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [I] Flink write iceberg bug(org.apache.iceberg.exceptions.NotFoundException) [iceberg]

2023-11-06 Thread via GitHub
pvary commented on issue #5846: URL: https://github.com/apache/iceberg/issues/5846#issuecomment-1794748179 @lirui-apache: Would you mind adding this to `docs/flink-writes.md`? I would be happy to review. Thanks, Peter -- This is an automated message from the Apache Git Service. To r

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383244841 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383232094 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383228989 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1383228406 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.21.10 to 2.21.15 [iceberg]

2023-11-06 Thread via GitHub
nastra merged PR #8983: URL: https://github.com/apache/iceberg/pull/8983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump arrow from 13.0.0 to 14.0.0 [iceberg]

2023-11-06 Thread via GitHub
nastra merged PR #8984: URL: https://github.com/apache/iceberg/pull/8984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Parquet: don't throw exception on row group filters when reading INT96 column [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8988: URL: https://github.com/apache/iceberg/pull/8988#discussion_r1383168479 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java: ## @@ -199,7 +201,7 @@ public Boolean lt(BoundReference ref, Literal lit) {

Re: [PR] Parquet: don't throw exception on row group filters when reading INT96 column [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8988: URL: https://github.com/apache/iceberg/pull/8988#discussion_r1383166362 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java: ## @@ -199,7 +201,7 @@ public Boolean lt(BoundReference ref, Literal lit) {

Re: [PR] Catch warning in PyLint tests [iceberg-python]

2023-11-06 Thread via GitHub
Fokko merged PR #33: URL: https://github.com/apache/iceberg-python/pull/33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Catch warning in PyLint tests [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on code in PR #33: URL: https://github.com/apache/iceberg-python/pull/33#discussion_r1383041054 ## pyiceberg/manifest.py: ## @@ -783,8 +783,8 @@ def __init__(self, spec: PartitionSpec, schema: Schema, output_file: OutputFile, output_file,

Re: [PR] Build: Bump mypy-boto3-glue from 1.28.63 to 1.28.77 [iceberg-python]

2023-11-06 Thread via GitHub
Fokko merged PR #130: URL: https://github.com/apache/iceberg-python/pull/130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Flink write iceberg bug(org.apache.iceberg.exceptions.NotFoundException) [iceberg]

2023-11-06 Thread via GitHub
lirui-apache commented on issue #5846: URL: https://github.com/apache/iceberg/issues/5846#issuecomment-1794403680 Hi @chenwyi2 @pvary , thanks for the clarifications. Yeah we changed our snapshot expire routine to keep the last snapshot created by Flink. If this is a limitation by design, I

Re: [I] support partition spec update in pyiceberg [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on issue #124: URL: https://github.com/apache/iceberg-python/issues/124#issuecomment-1794371184 Thanks for raising this! @puchengy which catalog are you using? Related is https://github.com/apache/iceberg-python/issues/22 -- This is an automated message from the Apa

Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-06 Thread via GitHub
Fokko commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1794369668 @HonahX Yes, I think that is how it should be done (Except the builder pattern, that's very much Java style :). I think we can split the work into several work packages: - Ab

Re: [I] Duplicate file name in Iceberg's metadata [iceberg]

2023-11-06 Thread via GitHub
github-raphael-douyere commented on issue #8953: URL: https://github.com/apache/iceberg/issues/8953#issuecomment-1794358478 We enabled S3 versioning on the bucket and can see a file name being used 2 times by 2 distincts micro-batches. So it is not a case of task retry inside Spark.

Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-06 Thread via GitHub
gaborkaszab commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1382965493 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify delete

[I] Remove properties from `JdbcUtil` [iceberg]

2023-11-06 Thread via GitHub
Fokko opened a new issue, #8989: URL: https://github.com/apache/iceberg/issues/8989 ### Feature Request / Improvement There are some duplicate properties: https://github.com/apache/iceberg/blob/b0bf62a448617bd5f57ca72c2648452e6600fa20/core/src/main/java/org/apache/iceberg/jdbc/JdbcUti

Re: [I] ICEBERG_CANNOT_OPEN_SPLIT: Error opening Iceberg split s3 [iceberg]

2023-11-06 Thread via GitHub
pawankukreja01 commented on issue #8427: URL: https://github.com/apache/iceberg/issues/8427#issuecomment-1794305805 This error occurs only in Athena, while running a query on the table using Spark works fine. According to the [Amazon EMR documentation](https://docs.aws.amazon.com/emr/latest

Re: [PR] Clarify time travel implementation in Iceberg [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1382915561 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2 table

Re: [PR] Clarify time travel implementation in Iceberg [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1382914816 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2 table

Re: [PR] Kafka Connect: Initial project setup and event data structures [iceberg]

2023-11-06 Thread via GitHub
nastra commented on code in PR #8701: URL: https://github.com/apache/iceberg/pull/8701#discussion_r1382910267 ## kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/AvroUtil.java: ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundati