[GitHub] [iceberg] rdblue merged pull request #6634: Core: Fix for deleting files when commiting transactions with multiple branches

rdblue merged PR #6634: URL: https://github.com/apache/iceberg/pull/6634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6634: Core: Fix for deleting files when commiting transactions with multiple branches

rdblue commented on PR #6634: URL: https://github.com/apache/iceberg/pull/6634#issuecomment-1400861489 Thanks, @amogh-jahagirdar! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

huaxingao commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1084457847 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.or

[GitHub] [iceberg] namrathamyske opened a new pull request, #6651: Spark 3.3 write to branch snapshot

namrathamyske opened a new pull request, #6651: URL: https://github.com/apache/iceberg/pull/6651 issue addressed from: https://github.com/apache/iceberg/issues/3896 Purpose of this PR is to enable spark writes to a branch snapshot. @rdblue @aokolnychyi @amogh-jahagirdar thought

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084484101 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -290,7 +293,12 @@ public String toString() { private class BatchApp

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084483714 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -304,4 +304,9 @@ public boolean caseSensitive() { .defaultValue(

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

namrathamyske commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084488656 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -304,4 +304,9 @@ public boolean caseSensitive() { .defaultValue(SQL

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

namrathamyske commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084488656 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -304,4 +304,9 @@ public boolean caseSensitive() { .defaultValue(SQL

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084500407 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -304,4 +304,9 @@ public boolean caseSensitive() { .defaultValue(

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1084502067 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceBranchExec.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to th

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084542063 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084546501 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084550316 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084551104 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084554897 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084556756 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] holdenk opened a new issue, #6652: Support for global shadow writes + logs

holdenk opened a new issue, #6652: URL: https://github.com/apache/iceberg/issues/6652 ### Feature Request / Improvement For job migration testing I'd really like to be able to set an environment variable that indicates all of the commits should be staged and logged for comparison.

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084554897 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084559034 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] aokolnychyi commented on pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on PR #6344: URL: https://github.com/apache/iceberg/pull/6344#issuecomment-1400987989 @flyrain, sorry it took me so long to get to this PR. Great work! I noticed a few things we should follow up on. Could you check? -- This is an automated message from the Apac

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1400990003 `0.13` should be fine. closing the issue. thanks @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

aokolnychyi commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1084563272 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [iceberg] dmgcodevil closed issue #6587: Wrong class, java.lang.Long, for object: 19367

dmgcodevil closed issue #6587: Wrong class, java.lang.Long, for object: 19367 URL: https://github.com/apache/iceberg/issues/6587 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

namrathamyske commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084571357 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -304,4 +304,9 @@ public boolean caseSensitive() { .defaultValue(SQL

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

aokolnychyi commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1084590470 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1084618789 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -82,6 +82,13 @@ snapshotRetentionC

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

amogh-jahagirdar commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1084620912 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala: ## @@ -38,33 +40,35 @@ case class

[GitHub] [iceberg] amogh-jahagirdar commented on issue #6647: Issue reading iceberg tables written by Athena with pyiceberg

amogh-jahagirdar commented on issue #6647: URL: https://github.com/apache/iceberg/issues/6647#issuecomment-1401099271 Need to investigate more deeply but based on the stack trace it seems like the metadata field on the parquet_schema is not even defined. Here's where pyarrow returns the sch

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

szehon-ho commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1084682564 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

szehon-ho commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1084698870 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] amogh-jahagirdar commented on issue #6647: Issue reading iceberg tables written by Athena with pyiceberg

amogh-jahagirdar commented on issue #6647: URL: https://github.com/apache/iceberg/issues/6647#issuecomment-1401198333 https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1022 Yeah the file key value metadata (which after stepping through the code) is

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

haizhou-zhao commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1084710334 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pr

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6654: Python: Check if optional file kv metadata is None before reading Iceberg Schema

amogh-jahagirdar opened a new pull request, #6654: URL: https://github.com/apache/iceberg/pull/6654 This is an interim solution for https://github.com/apache/iceberg/issues/6647. Parquet file k/v metadata is optional and not required to be written as per the Parquet spec https://github.com

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

haizhou-zhao commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1084710334 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pr

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

haizhou-zhao commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1084710334 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pr

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

haizhou-zhao commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1084710334 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pr

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

szehon-ho commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1084720439 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any props

[GitHub] [iceberg] szehon-ho commented on pull request #6591: Core: Avoid creating new metadata file when `registerTable` API is used

szehon-ho commented on PR #6591: URL: https://github.com/apache/iceberg/pull/6591#issuecomment-1401217397 Thanks @krvikash looks good to me, just wanted to see if we can add a test in TestHiveCatalog? (Ignore if you read already and still working on it) -- This is an automated message fr

[GitHub] [iceberg] aokolnychyi merged pull request #6633: Spark 3.3: Fix predicate pushdown for copy-on-write MERGE commands

aokolnychyi merged PR #6633: URL: https://github.com/apache/iceberg/pull/6633 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[GitHub] [iceberg] aokolnychyi commented on pull request #6633: Spark 3.3: Fix predicate pushdown for copy-on-write MERGE commands

aokolnychyi commented on PR #6633: URL: https://github.com/apache/iceberg/pull/6633#issuecomment-1401352111 Thank you, @amogh-jahagirdar @RussellSpitzer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1084790152 ## core/src/main/java/org/apache/iceberg/BasePositionDeletesScanTask.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1084792629 ## delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java: ## @@ -0,0 +1,468 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1084793195 ## delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java: ## @@ -0,0 +1,468 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] shardulm94 commented on pull request #6327: ORC: Fix error when projecting nested indentity partition column

shardulm94 commented on PR #6327: URL: https://github.com/apache/iceberg/pull/6327#issuecomment-1401365132 Hey @rdblue, can you take a look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1084795695 ## delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java: ## @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1084798607 ## delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java: ## @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1084802235 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1040,4 +1047,195 @@ public void testAllManifestsTableSnapshotNot() { expectedM

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1084805304 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,252 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

aokolnychyi commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1084807856 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6655: Spark: Handle ResolvingFileIO while determining LocalityPreference

amogh-jahagirdar commented on code in PR #6655: URL: https://github.com/apache/iceberg/pull/6655#discussion_r1084813006 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -67,11 +69,15 @@ public boolean caseSensitive() { } public boolean

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1084793195 ## delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java: ## @@ -0,0 +1,468 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] krvikash commented on pull request #6591: Core: Avoid creating new metadata file when `registerTable` API is used

krvikash commented on PR #6591: URL: https://github.com/apache/iceberg/pull/6591#issuecomment-1401425378 Thanks, @szehon-ho for reviewing. Sorry, I missed adding a test case in `TestHiveCatalog`. I have added it now. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [iceberg] ajantha-bhat commented on pull request #6656: Nessie: Avoid usage of deprecated APIs in test

ajantha-bhat commented on PR #6656: URL: https://github.com/apache/iceberg/pull/6656#issuecomment-1401434718 cc: @dimas-b -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [iceberg] ajantha-bhat closed pull request #3719: Spark: support delete_reachable_files procedure

ajantha-bhat closed pull request #3719: Spark: support delete_reachable_files procedure URL: https://github.com/apache/iceberg/pull/3719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] nastra commented on a diff in pull request #6562: Core: Improvements around Token Refresh time expiration

nastra commented on code in PR #6562: URL: https://github.com/apache/iceberg/pull/6562#discussion_r1084918753 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -550,31 +512,29 @@ public static AuthSession fromAccessToken( RESTClient client,

[GitHub] [iceberg] nastra commented on a diff in pull request #6562: Core: Improvements around Token Refresh time expiration

nastra commented on code in PR #6562: URL: https://github.com/apache/iceberg/pull/6562#discussion_r1084920594 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -611,17 +571,17 @@ private static AuthSession fromTokenResponse( AuthSession session =

[GitHub] [iceberg] deniskuzZ commented on pull request #6653: API: Fix Transform backward compatibility in PartitionSpec

deniskuzZ commented on PR #6653: URL: https://github.com/apache/iceberg/pull/6653#issuecomment-1401532219 @amogh-jahagirdar, thanks for checking! updated PR description and added unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [iceberg] Fokko opened a new pull request, #6657: Python: Allow to pass in a string as filter

Fokko opened a new pull request, #6657: URL: https://github.com/apache/iceberg/pull/6657 Often I have to look up the name of the operator, I think it would be nice to allow the end user to provide a string that we'll parse with the excellent parser that we already have. -- This is an aut

[GitHub] [iceberg] Fokko commented on a diff in pull request #6590: Python: Add sql command to the CLI

Fokko commented on code in PR #6590: URL: https://github.com/apache/iceberg/pull/6590#discussion_r1084938300 ## python/pyiceberg/cli/console.py: ## @@ -365,3 +366,37 @@ def table(ctx: Context, identifier: str, property_name: str) -> None: # noqa: F ctx.exit(1) el

[GitHub] [iceberg] nastra commented on a diff in pull request #6562: Core: Improvements around Token Refresh time expiration

nastra commented on code in PR #6562: URL: https://github.com/apache/iceberg/pull/6562#discussion_r1084938311 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -550,31 +512,29 @@ public static AuthSession fromAccessToken( RESTClient client,

[GitHub] [iceberg] nastra closed pull request #6562: Core: Improvements around Token Refresh time expiration

nastra closed pull request #6562: Core: Improvements around Token Refresh time expiration URL: https://github.com/apache/iceberg/pull/6562 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [iceberg] findepi commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

findepi commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1085085232 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.org/

[GitHub] [iceberg] sdaberdaku commented on issue #6615: Merge into does not work with spark temp table

sdaberdaku commented on issue #6615: URL: https://github.com/apache/iceberg/issues/6615#issuecomment-1401709057 I faced the same exact error as @alex2308 and tried a couple of things. There were no useful logs in the Spark UI, so I tried downgrading from the spark runtime from 3.3.1 to 3.2.

[GitHub] [iceberg-docs] Fokko merged pull request #195: Update to latest REST server image that uses Iceberg 1.1.0

Fokko merged PR #195: URL: https://github.com/apache/iceberg-docs/pull/195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[GitHub] [iceberg] Fokko commented on a diff in pull request #6590: Python: Add sql command to the CLI

Fokko commented on code in PR #6590: URL: https://github.com/apache/iceberg/pull/6590#discussion_r1085250404 ## python/pyiceberg/cli/console.py: ## @@ -365,3 +366,37 @@ def table(ctx: Context, identifier: str, property_name: str) -> None: # noqa: F ctx.exit(1) el

[GitHub] [iceberg] Fokko commented on pull request #6590: Python: Add sql command to the CLI

Fokko commented on PR #6590: URL: https://github.com/apache/iceberg/pull/6590#issuecomment-1401907191 I tested it locally and it works. Ran into some small issues with the serialization of datetimes: ``` pyiceberg --output json --catalog local sql --table nyc.taxis="tpep_pickup_dateti

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085315791 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveLock.java: ## @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085322064 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085348240 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085348865 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085350986 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085354225 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085356136 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on a diff in pull request #6648: Hive: Refactor commit lock mechanism from HiveTableOperations

pvary commented on code in PR #6648: URL: https://github.com/apache/iceberg/pull/6648#discussion_r1085379926 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] pvary commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

pvary commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1402028111 > static wrapper can make the object lifecycle management difficult. E.g., Flink needs to [unload dynamically loaded classes in user code](https://nightlies.apache.org/flink/flink-docs-mast

[GitHub] [iceberg] dimas-b commented on a diff in pull request #6656: Nessie: Avoid usage of deprecated APIs in test

dimas-b commented on code in PR #6656: URL: https://github.com/apache/iceberg/pull/6656#discussion_r1085396302 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -114,8 +113,8 @@ private void resetData() throws NessieConflictException, NessieNotFoundE

[GitHub] [iceberg] dimas-b commented on a diff in pull request #6656: Nessie: Avoid usage of deprecated APIs in test

dimas-b commented on code in PR #6656: URL: https://github.com/apache/iceberg/pull/6656#discussion_r1085396302 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -114,8 +113,8 @@ private void resetData() throws NessieConflictException, NessieNotFoundE

[GitHub] [iceberg] nastra commented on a diff in pull request #6649: Nessie-build: add test dependencies

nastra commented on code in PR #6649: URL: https://github.com/apache/iceberg/pull/6649#discussion_r1085444775 ## build.gradle: ## @@ -668,6 +668,8 @@ project(':iceberg-nessie') { implementation "com.fasterxml.jackson.core:jackson-core" testImplementation "org.project

[GitHub] [iceberg] amogh-jahagirdar closed issue #6632: Bug with Branch Transactions

amogh-jahagirdar closed issue #6632: Bug with Branch Transactions URL: https://github.com/apache/iceberg/issues/6632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [iceberg] amogh-jahagirdar commented on issue #6632: Bug with Branch Transactions

amogh-jahagirdar commented on issue #6632: URL: https://github.com/apache/iceberg/issues/6632#issuecomment-1402128554 Since the fix for this is merged, I'm going to close this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [iceberg] jackye1995 merged pull request #6358: AWS: Print logs whether Glue optimistic locking is used or not

jackye1995 merged PR #6358: URL: https://github.com/apache/iceberg/pull/6358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[GitHub] [iceberg] jackye1995 commented on pull request #6358: AWS: Print logs whether Glue optimistic locking is used or not

jackye1995 commented on PR #6358: URL: https://github.com/apache/iceberg/pull/6358#issuecomment-1402156201 Thanks for the fix @aajisaka , thanks for the reviews @rdblue and @amogh-jahagirdar ! -- This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [iceberg] jackye1995 commented on pull request #6623: AWS, Docs: Add AWS Glue in Run Iceberg on AWS section

jackye1995 commented on PR #6623: URL: https://github.com/apache/iceberg/pull/6623#issuecomment-1402161597 Congrats for the launch @aajisaka ! Thanks everyone for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [iceberg] jackye1995 merged pull request #6623: AWS, Docs: Add AWS Glue in Run Iceberg on AWS section

jackye1995 merged PR #6623: URL: https://github.com/apache/iceberg/pull/6623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085549065 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestGenerateChangesProcedure.java: ## @@ -0,0 +1,452 @@ +/* + * Licensed to the

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085571355 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085584919 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software

[GitHub] [iceberg] stevenzwu commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

stevenzwu commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1402252806 > What is the best way to use Connection pools in Flink tasks? Like a pool for HMSConnection, or JDBC driver? Shall we add that, and the wrapper classes to the main Flink classpath, s

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085603592 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software

[GitHub] [iceberg] stevenzwu commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

stevenzwu commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1402267342 > I think that if a catalog is closed, it's reasonable for tables to stop operating as well. The catalog manages its shared resources and if it chooses to share a connection pool with t

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085622525 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

RussellSpitzer commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1085623802 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,278 @@ +/* + * Licensed to the Apache Software

[GitHub] [iceberg] aokolnychyi commented on pull request #5615: Delete orphan files jmh benchmark

aokolnychyi commented on PR #5615: URL: https://github.com/apache/iceberg/pull/5615#issuecomment-1402298810 I will have to review today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [iceberg] aokolnychyi commented on pull request #6588: Spark 3.3: Add Default Parallelism Level for All Spark Driver Based Deletes

aokolnychyi commented on PR #6588: URL: https://github.com/apache/iceberg/pull/6588#issuecomment-1402302189 cc #5373 #5375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [iceberg] dchristle commented on issue #3703: DeleteOrphanFiles or ExpireSnapshots outofmemory

dchristle commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-1402362942 I'm following up to say I got `deleteOrphanFiles` to complete successfully. After bumping the memory, I was confused why I didn't see any output in the logs from an occasional `Ret

[GitHub] [iceberg] aajisaka commented on pull request #6358: AWS: Print logs whether Glue optimistic locking is used or not

aajisaka commented on PR #6358: URL: https://github.com/apache/iceberg/pull/6358#issuecomment-1402389457 Thank you for the reviews @rdblue @jackye1995 @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg] aajisaka commented on pull request #6623: AWS, Docs: Add AWS Glue in Run Iceberg on AWS section

aajisaka commented on PR #6623: URL: https://github.com/apache/iceberg/pull/6623#issuecomment-1402390300 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [iceberg] flyrain commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

flyrain commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1085728930 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.org/

[GitHub] [iceberg] Fokko commented on a diff in pull request #6654: Python: Check if optional Parquet kv metadata is None before reading Iceberg Schema

Fokko commented on code in PR #6654: URL: https://github.com/apache/iceberg/pull/6654#discussion_r1085731519 ## python/pyiceberg/io/pyarrow.py: ## @@ -505,7 +505,9 @@ def project_table( # Get the schema with fs.open_input_file(path) as fout: parque

[GitHub] [iceberg] rdblue commented on a diff in pull request #6562: Core: Improvements around Token Refresh time expiration

rdblue commented on code in PR #6562: URL: https://github.com/apache/iceberg/pull/6562#discussion_r1085735299 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -585,22 +547,24 @@ public static AuthSession fromAccessToken( long startTimeMillis = Sys

<    1   2   3   4   5   6   7   8   9   10   >