[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098064310 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098065548 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098066474 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098066977 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098067020 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098067598 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098067983 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098068922 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098069109 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -436,16 +436,14 @@ static TableMetadata fromJson(String metadataLocation, JsonNode node) {

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098071002 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1071,6 +1022,11 @@ public Builder addSortOrder(SortOrder order) { return this; } +p

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098072499 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098072879 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098073489 ## core/src/test/java/org/apache/iceberg/TestSnapshotOperations.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098074207 ## core/src/test/java/org/apache/iceberg/TestSnapshotOperations.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098075811 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098076045 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] 372242283 commented on issue #3466: "Manifest is missing" ValidationException when there have Concurrent applications to rewrite manifests

372242283 commented on issue #3466: URL: https://github.com/apache/iceberg/issues/3466#issuecomment-1420045743 > As I wrote above the issue os that the rewrite command becomes out of date while running so it fails. At least this is my hypothesis above This problem happened by accident

[GitHub] [iceberg] theoxu31 commented on a diff in pull request #6742: support registerTable in GlueCatalog

theoxu31 commented on code in PR #6742: URL: https://github.com/apache/iceberg/pull/6742#discussion_r1098092626 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java: ## @@ -437,6 +439,44 @@ public void renameTable(TableIdentifier from, TableIdentifier to) { LO

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098093494 ## build.gradle: ## @@ -438,6 +442,77 @@ project(':iceberg-aws') { } } +project(':iceberg-delta-lake') { + // use integration test since we can take advantages

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098093883 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098094220 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] zoucao commented on issue #6756: The out-of-order problem occurs around the process of recovery

zoucao commented on issue #6756: URL: https://github.com/apache/iceberg/issues/6756#issuecomment-1420050929 gentle ping @stevenzwu, what do you think about it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098094220 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098097479 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098097617 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,398 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098097479 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098097479 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

namrathamyske commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098126130 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStringM

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

namrathamyske commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098126130 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStringM

[GitHub] [iceberg] eric666666 commented on pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

eric66 commented on PR #6581: URL: https://github.com/apache/iceberg/pull/6581#issuecomment-1420127931 Is this action currently available? Is there any usage documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [iceberg] eric666666 commented on pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

eric66 commented on PR #6581: URL: https://github.com/apache/iceberg/pull/6581#issuecomment-1420131220 > Is this action currently available? Is there any usage documentation. We have already applied the v2 table to production. It is too uncomfortable to delete the delete file when

[GitHub] [iceberg] ludlows opened a new issue, #6759: Spark: use where condition to control the execution of rewriteDataFiles for procedure rewrite data files in spark sql

ludlows opened a new issue, #6759: URL: https://github.com/apache/iceberg/issues/6759 ### Feature Request / Improvement ### Improvement ### background we have known that the rewriteDataFiles is suggested to run periodically. in our production, we would like to run

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098137262 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/StreamingMonitorFunction.java: ## @@ -124,11 +126,33 @@ public void initializeState(FunctionInitial

[GitHub] [iceberg] ludlows opened a new pull request, #6760: spark: use a deterministic where condition to make rewrite_data_files…

ludlows opened a new pull request, #6760: URL: https://github.com/apache/iceberg/pull/6760 the behavior is requested in the issue #6759 here I implement the evaluation procedure to check if the where condition is a deterministic false. if so, the rewrite_data_files exits directly.

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

jackye1995 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098151208 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,398 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

jackye1995 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098151723 ## build.gradle: ## @@ -438,6 +442,77 @@ project(':iceberg-aws') { } } +project(':iceberg-delta-lake') { + // use integration test since we can take advantage

[GitHub] [iceberg] slfan1989 commented on pull request #6735: replace-parquet-deprecated-method

slfan1989 commented on PR #6735: URL: https://github.com/apache/iceberg/pull/6735#issuecomment-1420194127 @Fokko @jackye1995 Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [iceberg] bluzy opened a new pull request, #6761: Hive: add retry for metastore permission error

bluzy opened a new pull request, #6761: URL: https://github.com/apache/iceberg/pull/6761 FIx : https://github.com/apache/iceberg/issues/6750 ## Situation 1. A user create `RetryingMetaStoreClient` instance, use it, and release. 2. B user get `RetryingMetaStoreClient` created

[GitHub] [iceberg] pvary commented on a diff in pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

pvary commented on code in PR #6614: URL: https://github.com/apache/iceberg/pull/6614#discussion_r1098199612 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java: ## @@ -91,7 +106,6 @@ class CatalogTableLoader implements TableLoader { private final

[GitHub] [iceberg] pvary commented on a diff in pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

pvary commented on code in PR #6614: URL: https://github.com/apache/iceberg/pull/6614#discussion_r1098200427 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java: ## @@ -34,12 +34,16 @@ * the cluster (for example, to get splits), not just on the client

[GitHub] [iceberg] pvary commented on issue #6750: Failed to get table info from metastore using impersonation

pvary commented on issue #6750: URL: https://github.com/apache/iceberg/issues/6750#issuecomment-1420235524 I think that adding the user to the cache key in the HMS client pool would be the best solution. See #6175 -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [iceberg] pvary commented on pull request #6761: Hive: add retry for metastore permission error

pvary commented on PR #6761: URL: https://github.com/apache/iceberg/pull/6761#issuecomment-1420238961 I think #6175 would be a better fix for your issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] bluzy closed pull request #6761: Hive: add retry for metastore permission error

bluzy closed pull request #6761: Hive: add retry for metastore permission error URL: https://github.com/apache/iceberg/pull/6761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [iceberg] bluzy commented on pull request #6761: Hive: add retry for metastore permission error

bluzy commented on PR #6761: URL: https://github.com/apache/iceberg/pull/6761#issuecomment-1420241011 > I think #6175 would be a better fix for your issue @pvary Oh I didn't know that. I think #6175 can solve my issue. Thank you. -- This is an automated message from the Apache Gi

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098210476 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,398 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1098210476 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,398 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097921035 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Softw

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

hililiwei commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098223331 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -210,7 +210,9 @@ class Iceb

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

hililiwei commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098223331 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -210,7 +210,9 @@ class Iceb

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

hililiwei commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098223331 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -210,7 +210,9 @@ class Iceb

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

hililiwei commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098224917 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | ALTER TAB

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098226674 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | AL

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098226674 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | AL

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098227272 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Softw

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098228124 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Softw

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098229352 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -210,7 +210,9 @@ cla

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1098239129 ## api/src/main/java/org/apache/iceberg/expressions/BoundAggregate.java: ## @@ -44,4 +57,85 @@ public Type type() { return term().type(); } } + + publ

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1098240039 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1098240584 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098252250 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -125,7 +137,24 @@ private void validate() { startSnapshotId ==

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098252250 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -125,7 +137,24 @@ private void validate() { startSnapshotId ==

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098266653 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -125,7 +137,24 @@ private void validate() { startSnapshotId ==

[GitHub] [iceberg] chenjunjiedada commented on pull request #6753: Flink: use projection when delete records in upsert mode

chenjunjiedada commented on PR #6753: URL: https://github.com/apache/iceberg/pull/6753#issuecomment-1420333684 @stevenzwu @hililiwei You guys may want to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098292696 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | AL

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098266653 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -125,7 +137,24 @@ private void validate() { startSnapshotId ==

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1098266653 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -125,7 +137,24 @@ private void validate() { startSnapshotId ==

[GitHub] [iceberg] Fokko commented on a diff in pull request #6646: Implement Support for DynamoDB Catalog

Fokko commented on code in PR #6646: URL: https://github.com/apache/iceberg/pull/6646#discussion_r1098384159 ## python/pyiceberg/catalog/dynamodb.py: ## @@ -0,0 +1,776 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [iceberg] tprelle commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

tprelle commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1098436510 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] tprelle commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

tprelle commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1098436804 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] tprelle commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

tprelle commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1098438966 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java: ## @@ -448,4 +448,37 @@ public void testExtraSnapshotMetadataWithSQL() thr

[GitHub] [iceberg] nastra commented on a diff in pull request #6757: API, Core: Multi-Table transactions

nastra commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1098466518 ## api/src/main/java/org/apache/iceberg/catalog/CatalogTransaction.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] nastra commented on a diff in pull request #6757: API, Core: Multi-Table transactions

nastra commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1098467113 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalogTransaction.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] Akshay-connectwise commented on issue #2040: Partial data ingestion to Iceberg in failing with Spark 3.0.x

Akshay-connectwise commented on issue #2040: URL: https://github.com/apache/iceberg/issues/2040#issuecomment-1420569509 Hi @RussellSpitzer, Have tried setting up the table properties but it didn't worked for me below is table creation query and error when I have an extra column "comment"

[GitHub] [iceberg] nastra commented on a diff in pull request #6757: API, Core: Multi-Table transactions

nastra commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1098466518 ## api/src/main/java/org/apache/iceberg/catalog/CatalogTransaction.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] nastra commented on a diff in pull request #6757: API, Core: Multi-Table transactions

nastra commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1098467113 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalogTransaction.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] nastra commented on a diff in pull request #6757: API, Core: Multi-Table transactions

nastra commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1098467113 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalogTransaction.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] RussellSpitzer commented on issue #2040: Partial data ingestion to Iceberg in failing with Spark 3.0.x

RussellSpitzer commented on issue #2040: URL: https://github.com/apache/iceberg/issues/2040#issuecomment-1420792180 The property key is 'write.spark.accept-any-schema' And the value should be 'true' -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [iceberg] Akshay-connectwise commented on issue #2040: Partial data ingestion to Iceberg in failing with Spark 3.0.x

Akshay-connectwise commented on issue #2040: URL: https://github.com/apache/iceberg/issues/2040#issuecomment-1420826427 Thanks!! After adding the property key able to append data with additional column while keeping "mergeSchema" option true. -- This is an automated message from the Ap

[GitHub] [iceberg] munendrasn opened a new issue, #6763: ACL when using DynamoDb based Catalog

munendrasn opened a new issue, #6763: URL: https://github.com/apache/iceberg/issues/6763 ### Query engine _No response_ ### Question We are using the DynamoDb based Catalog implementation, and working on providing namespace level access control. DynamoDb table per

[GitHub] [iceberg] fqaiser94 commented on a diff in pull request #6513: API: Support Conditional Commits - WIP

fqaiser94 commented on code in PR #6513: URL: https://github.com/apache/iceberg/pull/6513#discussion_r1098754537 ## api/src/main/java/org/apache/iceberg/ValidatablePendingUpdate.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] fqaiser94 commented on issue #6514: Support Conditional Transaction Commits

fqaiser94 commented on issue #6514: URL: https://github.com/apache/iceberg/issues/6514#issuecomment-1420971731 Created a PR for this here: https://github.com/apache/iceberg/pull/6513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [iceberg] rdblue merged pull request #6738: Python: Bump pre-commit plugins to the latest version

rdblue merged PR #6738: URL: https://github.com/apache/iceberg/pull/6738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6738: Python: Bump pre-commit plugins to the latest version

rdblue commented on PR #6738: URL: https://github.com/apache/iceberg/pull/6738#issuecomment-1421053585 Thanks, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [iceberg] rdblue merged pull request #6737: Python: Avoid local imports in test

rdblue merged PR #6737: URL: https://github.com/apache/iceberg/pull/6737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue merged pull request #6719: Python: Publish the docs by hand

rdblue merged PR #6719: URL: https://github.com/apache/iceberg/pull/6719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6719: Python: Publish the docs by hand

rdblue commented on PR #6719: URL: https://github.com/apache/iceberg/pull/6719#issuecomment-1421059012 Sorry, @Fokko. I thought I had reviewed and merged this one already, but it looks like I hadn't finished it. -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [iceberg] rdblue merged pull request #6657: Python: Allow to pass in a string as filter

rdblue merged PR #6657: URL: https://github.com/apache/iceberg/pull/6657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6657: Python: Allow to pass in a string as filter

rdblue commented on PR #6657: URL: https://github.com/apache/iceberg/pull/6657#issuecomment-1421061183 Thanks for the update, @Fokko! Looks good now so I merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] nastra commented on a diff in pull request #6706: Refactor table metadata snapshot management

nastra commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098892936 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] jackye1995 commented on pull request #6717: spark 3.3 read by snapshot ref schema

jackye1995 commented on PR #6717: URL: https://github.com/apache/iceberg/pull/6717#issuecomment-1421091541 @aokolnychyi you might be also interested in this because #6651 depends on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1098935911 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | AL

[GitHub] [iceberg] nastra commented on a diff in pull request #6706: Refactor table metadata snapshot management

nastra commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098938112 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

RussellSpitzer commented on code in PR #6554: URL: https://github.com/apache/iceberg/pull/6554#discussion_r1098942530 ## data/src/test/java/org/apache/iceberg/data/TestMetricsRowGroupFilter.java: ## @@ -86,6 +65,27 @@ import org.junit.runner.RunWith; import org.junit.runners.P

[GitHub] [iceberg] Fokko commented on pull request #6644: Python: Add support for static table

Fokko commented on PR #6644: URL: https://github.com/apache/iceberg/pull/6644#issuecomment-1421118698 @rubenvdg can you rebase? We were waiting for https://github.com/apache/iceberg/pull/6719 to be merged so we don't update the docs right away -- This is an automated message from the Apa

<    10   11   12   13   14   15   16   17   18   19   >