Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1398110471 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,359 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1398110471 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,359 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1398107792 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.proj

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1398105061 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,177 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1398104467 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software

Re: [I] Spark: Extend expire_snapshots procedure with an optional arg for snapshot ids [iceberg]

aokolnychyi commented on issue #2221: URL: https://github.com/apache/iceberg/issues/2221#issuecomment-1817388782 Resolving as not something we want to do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Spark: Extend expire_snapshots procedure with an optional arg for snapshot ids [iceberg]

aokolnychyi closed issue #2221: Spark: Extend expire_snapshots procedure with an optional arg for snapshot ids URL: https://github.com/apache/iceberg/issues/2221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Support vectorized reads with delete files [iceberg]

aokolnychyi closed issue #2247: Support vectorized reads with delete files URL: https://github.com/apache/iceberg/issues/2247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Support vectorized reads with delete files [iceberg]

aokolnychyi commented on issue #2247: URL: https://github.com/apache/iceberg/issues/2247#issuecomment-1817388018 This is done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Spark SQL Extensions: Handle concurrent modifications on same table instance [iceberg]

aokolnychyi commented on issue #1768: URL: https://github.com/apache/iceberg/issues/1768#issuecomment-1817387484 This is done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Spark SQL Extensions: Handle concurrent modifications on same table instance [iceberg]

aokolnychyi closed issue #1768: Spark SQL Extensions: Handle concurrent modifications on same table instance URL: https://github.com/apache/iceberg/issues/1768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Spark SQL Extensions: Add proper logging [iceberg]

aokolnychyi commented on issue #1889: URL: https://github.com/apache/iceberg/issues/1889#issuecomment-1817387439 This has been done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Create an action to clean up tables [iceberg]

aokolnychyi commented on issue #1549: URL: https://github.com/apache/iceberg/issues/1549#issuecomment-1817387035 This has been done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Create an action to clean up tables [iceberg]

aokolnychyi closed issue #1549: Create an action to clean up tables URL: https://github.com/apache/iceberg/issues/1549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [I] Consider defaulting PartitionSpec and SortOrder if they are null [iceberg]

aokolnychyi commented on issue #1411: URL: https://github.com/apache/iceberg/issues/1411#issuecomment-1817386881 This has been done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Consider defaulting PartitionSpec and SortOrder if they are null [iceberg]

aokolnychyi closed issue #1411: Consider defaulting PartitionSpec and SortOrder if they are null URL: https://github.com/apache/iceberg/issues/1411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Expose time travelling using multi-part identifiers [iceberg]

aokolnychyi commented on issue #1260: URL: https://github.com/apache/iceberg/issues/1260#issuecomment-1817386799 There are native APIs in Spark now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Expose time travelling using multi-part identifiers [iceberg]

aokolnychyi closed issue #1260: Expose time travelling using multi-part identifiers URL: https://github.com/apache/iceberg/issues/1260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Investigate classes that use Guava and may be exposed to serialization by Spark [iceberg]

aokolnychyi closed issue #1337: Investigate classes that use Guava and may be exposed to serialization by Spark URL: https://github.com/apache/iceberg/issues/1337 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] Investigate classes that use Guava and may be exposed to serialization by Spark [iceberg]

aokolnychyi commented on issue #1337: URL: https://github.com/apache/iceberg/issues/1337#issuecomment-1817386703 This has been resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Support vectorized reads with identity transforms [iceberg]

aokolnychyi commented on issue #1257: URL: https://github.com/apache/iceberg/issues/1257#issuecomment-1817386533 This has been resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Support vectorized reads with identity transforms [iceberg]

aokolnychyi closed issue #1257: Support vectorized reads with identity transforms URL: https://github.com/apache/iceberg/issues/1257 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Fix hadoop data source options in Spark [iceberg]

aokolnychyi closed issue #259: Fix hadoop data source options in Spark URL: https://github.com/apache/iceberg/issues/259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Fix hadoop data source options in Spark [iceberg]

aokolnychyi commented on issue #259: URL: https://github.com/apache/iceberg/issues/259#issuecomment-1817386281 No longer applies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Codahale instrumentation [iceberg]

aokolnychyi commented on issue #617: URL: https://github.com/apache/iceberg/issues/617#issuecomment-1817386155 We have ScanMetrics API now. Closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Codahale instrumentation [iceberg]

aokolnychyi closed issue #617: Codahale instrumentation URL: https://github.com/apache/iceberg/issues/617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] Accumulate requirements for format compatibility v2 [iceberg]

aokolnychyi commented on issue #773: URL: https://github.com/apache/iceberg/issues/773#issuecomment-1817386019 The V2 spec has been voted on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Accumulate requirements for format compatibility v2 [iceberg]

aokolnychyi closed issue #773: Accumulate requirements for format compatibility v2 URL: https://github.com/apache/iceberg/issues/773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Consider reducing the number of shuffle partitions in Spark tests [iceberg]

aokolnychyi commented on issue #948: URL: https://github.com/apache/iceberg/issues/948#issuecomment-1817385958 This has beed done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Consider reducing the number of shuffle partitions in Spark tests [iceberg]

aokolnychyi closed issue #948: Consider reducing the number of shuffle partitions in Spark tests URL: https://github.com/apache/iceberg/issues/948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Document release candidate validation process [iceberg]

aokolnychyi commented on issue #568: URL: https://github.com/apache/iceberg/issues/568#issuecomment-1817385805 This has been resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Document release candidate validation process [iceberg]

aokolnychyi closed issue #568: Document release candidate validation process URL: https://github.com/apache/iceberg/issues/568 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Use Iceberg tables as sources for Spark Structured Streaming [iceberg]

aokolnychyi commented on issue #179: URL: https://github.com/apache/iceberg/issues/179#issuecomment-1817335040 Yes, this has been resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Use Iceberg tables as sources for Spark Structured Streaming [iceberg]

aokolnychyi closed issue #179: Use Iceberg tables as sources for Spark Structured Streaming URL: https://github.com/apache/iceberg/issues/179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Spark 3.5: Extend action for rewriting manifests to support deletes [iceberg]

aokolnychyi commented on PR #9020: URL: https://github.com/apache/iceberg/pull/9020#issuecomment-1817334731 Thank you for reviewing, @ajantha-bhat @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Consider delete manifests for rewrite manifests [iceberg]

aokolnychyi closed issue #6375: Consider delete manifests for rewrite manifests URL: https://github.com/apache/iceberg/issues/6375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Spark 3.5: Extend action for rewriting manifests to support deletes [iceberg]

aokolnychyi merged PR #9020: URL: https://github.com/apache/iceberg/pull/9020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] test: Rest catalog integration test. [iceberg-rust]

liurenjie1024 commented on issue #100: URL: https://github.com/apache/iceberg-rust/issues/100#issuecomment-1817321177 I'm happy to take this, I have implemented similar things in icelake. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark 3.4 3.3: Support metadata columns in staged scan (#8872) [iceberg]

aokolnychyi commented on PR #9098: URL: https://github.com/apache/iceberg/pull/9098#issuecomment-1817309535 Thank you, @zinking! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Spark 3.4 3.3: Support metadata columns in staged scan (#8872) [iceberg]

aokolnychyi merged PR #9098: URL: https://github.com/apache/iceberg/pull/9098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark: Add spark.sql.iceberg.locality to control whether fetch locali… [iceberg]

aokolnychyi commented on code in PR #9101: URL: https://github.com/apache/iceberg/pull/9101#discussion_r1398026346 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -63,4 +63,7 @@ private SparkSQLProperties() {} // Controls the WAP bran

Re: [PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar commented on code in PR #9102: URL: https://github.com/apache/iceberg/pull/9102#discussion_r1398015111 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -327,12 +331,9 @@ private boolean canDeleteUsingMetadata(Expression d

Re: [PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar commented on code in PR #9102: URL: https://github.com/apache/iceberg/pull/9102#discussion_r1398015111 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -327,12 +331,9 @@ private boolean canDeleteUsingMetadata(Expression d

Re: [PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar commented on code in PR #9102: URL: https://github.com/apache/iceberg/pull/9102#discussion_r1398013059 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -326,9 +326,8 @@ public void testDeleteWithoutScanningT

Re: [PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar commented on code in PR #9102: URL: https://github.com/apache/iceberg/pull/9102#discussion_r1398013059 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -326,9 +326,8 @@ public void testDeleteWithoutScanningT

Re: [PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar commented on code in PR #9102: URL: https://github.com/apache/iceberg/pull/9102#discussion_r1398012904 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -135,7 +135,7 @@ public static Obj

[PR] Spark: Fix metadata delete condition check when there are branches [iceberg]

amogh-jahagirdar opened a new pull request, #9102: URL: https://github.com/apache/iceberg/pull/9102 Fixes #7635 . The Iceberg Spark integration has logic to surface to the Spark Optimizer whether a metadata delete (deleting a data file) should be performed or not. When there are bran

Re: [I] Pick split size automatically [iceberg]

github-actions[bot] commented on issue #7465: URL: https://github.com/apache/iceberg/issues/7465#issuecomment-1817275291 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Most efficient way to "upsert" data into an Iceberg database [iceberg]

github-actions[bot] closed issue #7492: Most efficient way to "upsert" data into an Iceberg database URL: https://github.com/apache/iceberg/issues/7492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Pick split size automatically [iceberg]

github-actions[bot] closed issue #7465: Pick split size automatically URL: https://github.com/apache/iceberg/issues/7465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397782440 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397782440 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

Re: [I] Failure to delete with merge-on-read in branch and main branch [iceberg]

rdblue commented on issue #7635: URL: https://github.com/apache/iceberg/issues/7635#issuecomment-1817116426 @amogh-jahagirdar, that looks right to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] java.lang.IllegalStateException: Connection pool shut down when refreshing table metadata on s3 [iceberg]

mas-chen commented on issue #8601: URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1816994950 > It does not seem to be the same problem. According to the stacktrace, `Connection pool shut down` was not raised by an STS client. There could be some other problems with the life

Re: [I] Support relative paths in Table Metadata [iceberg]

jotarada commented on issue #1617: URL: https://github.com/apache/iceberg/issues/1617#issuecomment-1816946130 @abmo-x Can i do that on trino as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark: Add spark.sql.iceberg.locality to control whether fetch locali… [iceberg]

aokolnychyi commented on PR #9101: URL: https://github.com/apache/iceberg/pull/9101#issuecomment-1816936129 I will check later today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Failure to delete with merge-on-read in branch and main branch [iceberg]

amogh-jahagirdar commented on issue #7635: URL: https://github.com/apache/iceberg/issues/7635#issuecomment-1816926262 Ok I think got the root cause, the problem is in `canDeleteUsingMetadata` https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397721229 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397716578 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -429,6 +444,30 @@ public Builder setAll(Map properties) { return t

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397558768 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,215 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397557645 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,215 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397557005 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/WatermarkExtractorRecordEmitter.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Use StringUtils to verify null and empty string [iceberg]

nastra commented on code in PR #9090: URL: https://github.com/apache/iceberg/pull/9090#discussion_r1397540875 ## aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbCatalog.java: ## @@ -130,7 +130,7 @@ public void initialize(String name, Map properties) { void initiali

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397517633 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397517358 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/TimestampBasedWatermarkExtractor.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397486522 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergWatermarkExtractor.java: ## @@ -18,19 +18,11 @@ */ package org.apache.iceberg.flink.so

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397485790 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,377 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Use StringUtils to verify null and empty string [iceberg]

ndrluis commented on code in PR #9090: URL: https://github.com/apache/iceberg/pull/9090#discussion_r1397473471 ## aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbCatalog.java: ## @@ -130,7 +130,7 @@ public void initialize(String name, Map properties) { void initial

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

syun64 commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1397367899 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -370,7 +369,7 @@ public void testCatalogCredential() { Mockito.verify(adapter) .e

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

nastra commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1397361694 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -370,7 +369,7 @@ public void testCatalogCredential() { Mockito.verify(adapter) .e

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

snazy commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1397204876 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -132,74 +130,42 @@ protected void doRefresh() { @Override protected void doCom

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

nastra merged PR #9061: URL: https://github.com/apache/iceberg/pull/9061 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

nastra commented on PR #8909: URL: https://github.com/apache/iceberg/pull/8909#issuecomment-1816468831 > Lots of the existing code for tables is now duplicated for views - the critical code paths should be unified. I would suggest to do those refactorings as a separate PR, otherwise

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1397340519 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

snazy commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1397151718 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -312,26 +327,32 @@ namespace, getRef().getName()), } public void renameTable(Tab

Re: [PR] Nessie: reimplement namespace operations [iceberg]

snazy commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1816194902 > We would also be missing the commit meta specificities, which is the primary reason for this change. That functionality [exists in the client-side implementation](https://github.co

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397087181 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397085585 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Softw

[PR] Spark: Add spark.sql.iceberg.locality to control whether fetch locali… [iceberg]

zhongyujiang opened a new pull request, #9101: URL: https://github.com/apache/iceberg/pull/9101 …ty information while planing. We have a computing cluster which is seperate from our HDFS cluster, jobs running on this cluster will never be able to take advantage of the HDFS locality i

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1397050299 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache Softw

Re: [I] Unable to find a jdbc driver in a Flink execution environment. [iceberg]

JuiceFV closed issue #9097: Unable to find a jdbc driver in a Flink execution environment. URL: https://github.com/apache/iceberg/issues/9097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Unable to find a jdbc driver in a Flink execution environment. [iceberg]

JuiceFV commented on issue #9097: URL: https://github.com/apache/iceberg/issues/9097#issuecomment-1816076061 @nastra Yes, you're right, it's not an iceberg issue. I've finally found [it](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#unloading

Re: [PR] Nessie: reimplement namespace operations [iceberg]

adutra commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1815978613 > I'd prefer to avoid duplicating the functionality that already exists in `org.projectnessie.client.api.ns.ClientSide*Namespace`. That is not at all what was decided [here](https:

Re: [I] It sometimes throws exception java.lang.AssertionError: assertion failed after upgrade to Iceberg 1.3.1 + Spark 3.4.1 [iceberg]

jiantao-vungle commented on issue #9092: URL: https://github.com/apache/iceberg/issues/9092#issuecomment-1815955017 @nastra no very special operations inside this module, except that we have a deprecated field named `timestamp__deprecated` and we will insert value `NULL` for it. -- This

Re: [I] hive iceberg [iceberg]

luxl6 commented on issue #9094: URL: https://github.com/apache/iceberg/issues/9094#issuecomment-1815919122 Hello, I am using hive in CDH, version 2.1.1. Does it not support iceberg?? Writing data is in the temporary table, and data migration is not feasible. It has been in the temporary tab

Re: [I] It sometimes throws exception java.lang.AssertionError: assertion failed after upgrade to Iceberg 1.3.1 + Spark 3.4.1 [iceberg]

jackwang2 commented on issue #9092: URL: https://github.com/apache/iceberg/issues/9092#issuecomment-1815909030 Add @jiantao-vungle and @daoxunwu-vungle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark 3.5: Extend action for rewriting manifests to support deletes [iceberg]

nastra commented on code in PR #9020: URL: https://github.com/apache/iceberg/pull/9020#discussion_r1396839286 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -649,6 +659,249 @@ public void testRewriteLargeManifestsEvolve

Re: [I] It sometimes throws exception java.lang.AssertionError: assertion failed after upgrade to Iceberg 1.3.1 + Spark 3.4.1 [iceberg]

nastra commented on issue #9092: URL: https://github.com/apache/iceberg/issues/9092#issuecomment-1815898901 @jackwang2 I didn't mean the SQL but the actual code that's inside those custom modules. You must be doing something that fails that particular assertion you mentioned above. -- Th