Re: [I] struct value design [iceberg-rust]

2023-11-15 Thread via GitHub
liurenjie1024 commented on issue #77: URL: https://github.com/apache/iceberg-rust/issues/77#issuecomment-1813893291 Tracked in #96 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] struct value design [iceberg-rust]

2023-11-15 Thread via GitHub
liurenjie1024 closed issue #77: struct value design URL: https://github.com/apache/iceberg-rust/issues/77 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] Unable to append to table after cancelled optmization procedure call [iceberg]

2023-11-15 Thread via GitHub
LucasRoesler closed issue #7657: Unable to append to table after cancelled optmization procedure call URL: https://github.com/apache/iceberg/issues/7657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Unable to append to table after cancelled optmization procedure call [iceberg]

2023-11-15 Thread via GitHub
LucasRoesler commented on issue #7657: URL: https://github.com/apache/iceberg/issues/7657#issuecomment-1813875337 I haven't been able to reproduce this since it happened. I will close it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[I] Metrics for Manifest file caching [iceberg]

2023-11-15 Thread via GitHub
agrawalreetika opened a new issue, #9093: URL: https://github.com/apache/iceberg/issues/9093 ### Query engine Presto ### Question I was also exploring the Manifest file caching feature which was introduced in Iceberg https://github.com/apache/iceberg/pull/4518 for the Pr

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1395154710 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache S

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8872: URL: https://github.com/apache/iceberg/pull/8872#issuecomment-1813787907 Thank you, @zinking! Would you be interested to cherry-pick this change to other Spark versions? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi merged PR #8872: URL: https://github.com/apache/iceberg/pull/8872 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Add log entry for bloom filter [iceberg]

2023-11-15 Thread via GitHub
huaxingao commented on code in PR #9010: URL: https://github.com/apache/iceberg/pull/9010#discussion_r1395125605 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java: ## @@ -114,10 +119,17 @@ private boolean eval( Set filterRefs =

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8872: URL: https://github.com/apache/iceberg/pull/8872#issuecomment-1813689417 One minor comment and should be good to go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8872: URL: https://github.com/apache/iceberg/pull/8872#discussion_r1395073725 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScan.java: ## @@ -77,14 +77,16 @@ public boolean equals(Object other) { SparkStage

Re: [PR] Hive: Improve performance of HiveCatalog#listTables [iceberg]

2023-11-15 Thread via GitHub
link3280 commented on PR #9032: URL: https://github.com/apache/iceberg/pull/9032#issuecomment-1813672107 Closed for duplication of #2722. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Hive: Improve performance of HiveCatalog#listTables [iceberg]

2023-11-15 Thread via GitHub
link3280 closed pull request #9032: Hive: Improve performance of HiveCatalog#listTables URL: https://github.com/apache/iceberg/pull/9032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Hive: Improve performance of HiveCatalog#listTables [iceberg]

2023-11-15 Thread via GitHub
link3280 closed issue #9031: Hive: Improve performance of HiveCatalog#listTables URL: https://github.com/apache/iceberg/issues/9031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] Hive: Improve performance of HiveCatalog#listTables [iceberg]

2023-11-15 Thread via GitHub
link3280 commented on issue #9031: URL: https://github.com/apache/iceberg/issues/9031#issuecomment-1813669830 @pvary Thanks for the pointer! Then I'll close this issue in favor of the other. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] java.lang.IllegalStateException: Connection pool shut down when refreshing table metadata on s3 [iceberg]

2023-11-15 Thread via GitHub
Kontinuation commented on issue #8601: URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1813647919 It does not seem to be the same problem. According to the stacktrace, `Connection pool shut down` was not raised by an STS client. There could be some other problems with the li

Re: [PR] Core: Disallow setting equality field IDs for data [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8970: URL: https://github.com/apache/iceberg/pull/8970#discussion_r1395060541 ## core/src/main/java/org/apache/iceberg/DataFiles.java: ## @@ -301,12 +299,10 @@ public Builder withSplitOffsets(List offsets) { return this; } +

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1394590199 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/TimestampBasedWatermarkExtractor.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache So

[I] Remove `datatype()` method in `Literal`. [iceberg-rust]

2023-11-15 Thread via GitHub
liurenjie1024 opened a new issue, #95: URL: https://github.com/apache/iceberg-rust/issues/95 Per [discussion](https://github.com/apache/iceberg-rust/pull/82#discussion_r1392079536), we should remove [this method](https://github.com/apache/iceberg-rust/blob/28d7006ba3af12b1f99dc078780b08cc

Re: [PR] Add log entry for bloom filter [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #9010: URL: https://github.com/apache/iceberg/pull/9010#discussion_r1395045414 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java: ## @@ -114,10 +119,17 @@ private boolean eval( Set filterRefs =

Re: [PR] Add log entry for bloom filter [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #9010: URL: https://github.com/apache/iceberg/pull/9010#discussion_r1395044921 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java: ## @@ -114,10 +119,17 @@ private boolean eval( Set filterRefs =

Re: [PR] Add log entry for bloom filter [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #9010: URL: https://github.com/apache/iceberg/pull/9010#discussion_r1395044921 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java: ## @@ -114,10 +119,17 @@ private boolean eval( Set filterRefs =

Re: [PR] Add log entry for bloom filter [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #9010: URL: https://github.com/apache/iceberg/pull/9010#discussion_r1395044176 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java: ## @@ -114,10 +119,17 @@ private boolean eval( Set filterRefs =

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1813526621 I am not sure I entirely understood how `SerializableTable` instances are created in your case. Is it created from the primary table before sending the table info to remote nodes?

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8872: URL: https://github.com/apache/iceberg/pull/8872#discussion_r1394989386 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScanBuilder.java: ## @@ -18,27 +18,76 @@ */ package org.apache.iceberg.spark.source;

[I] Extend check-nullability parameter scope to allow writing optional list elements map values to required elements and values [iceberg]

2023-11-15 Thread via GitHub
Aulust opened a new issue, #9091: URL: https://github.com/apache/iceberg/issues/9091 ### Feature Request / Improvement The problem is the same as has been discussed in this issue https://github.com/apache/iceberg/issues/510 and addressed in this pull request https://github.com/apache

Re: [I] delete files with spark sql failed [iceberg]

2023-11-15 Thread via GitHub
github-actions[bot] closed issue #7522: delete files with spark sql failed URL: https://github.com/apache/iceberg/issues/7522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] delete files with spark sql failed [iceberg]

2023-11-15 Thread via GitHub
github-actions[bot] commented on issue #7522: URL: https://github.com/apache/iceberg/issues/7522#issuecomment-1813498657 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Iceberg with Hive Metastore does not create a catalog in Spark and uses default [iceberg]

2023-11-15 Thread via GitHub
github-actions[bot] commented on issue #7574: URL: https://github.com/apache/iceberg/issues/7574#issuecomment-1813498581 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Unable to append to table after cancelled optmization procedure call [iceberg]

2023-11-15 Thread via GitHub
github-actions[bot] commented on issue #7657: URL: https://github.com/apache/iceberg/issues/7657#issuecomment-1813498512 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Failure to delete with merge-on-read in branch and main branch [iceberg]

2023-11-15 Thread via GitHub
amogh-jahagirdar commented on issue #7635: URL: https://github.com/apache/iceberg/issues/7635#issuecomment-1813498341 Also, I should note that I can repro this only when there's the `COALESCE` query hint and it gets applied. When there's no `COALESCE` and there's separate files the delete w

Re: [PR] fix: Migrate from tempdir to tempfile crate [iceberg-rust]

2023-11-15 Thread via GitHub
Fokko commented on PR #91: URL: https://github.com/apache/iceberg-rust/pull/91#issuecomment-1813479844 Thanks @cdaudt for fixing this, and @liurenjie1024 & @Xuanwo for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] fix: Migrate from tempdir to tempfile crate [iceberg-rust]

2023-11-15 Thread via GitHub
Fokko merged PR #91: URL: https://github.com/apache/iceberg-rust/pull/91 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] fix: Migrate from tempdir to tempfile crate [iceberg-rust]

2023-11-15 Thread via GitHub
liurenjie1024 commented on PR #91: URL: https://github.com/apache/iceberg-rust/pull/91#issuecomment-1813467150 CC @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

2023-11-15 Thread via GitHub
Fokko commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-1813465479 > do we need any changes in readers to benefit from this? If not, can we run some existing benchmarks to showcase the read improvement is as we anticipate? Since we use the decoders f

Re: [PR] support read meta columns in staged scan [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8872: URL: https://github.com/apache/iceberg/pull/8872#issuecomment-1813458768 I should be able to take a look this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-1813448646 Also, nice work on a new encoder in Avro, @Fokko! Do you know when will that be available? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-1813420551 Question. Aren't we are using `DataFileWriter` from Avro in our `AvroFileAppender`? If so, how is this PR affecting it? Won't we still use direct encoders there? https://githu

Re: [PR] Use !.isEmpty() instead of .length() > 0 [iceberg]

2023-11-15 Thread via GitHub
Fokko commented on code in PR #9090: URL: https://github.com/apache/iceberg/pull/9090#discussion_r1394956912 ## spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -212,7 +212,7 @@ protected void createOrRepl

Re: [PR] Use !.isEmpty() instead of .length() > 0 [iceberg]

2023-11-15 Thread via GitHub
ndrluis commented on PR #9090: URL: https://github.com/apache/iceberg/pull/9090#issuecomment-1813399577 Done! @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Update table metadata [iceberg-python]

2023-11-15 Thread via GitHub
Fokko commented on PR #139: URL: https://github.com/apache/iceberg-python/pull/139#issuecomment-1813398715 @HonahX This is high on my list. I'm OOO the rest of the week, I'll review this early next week since this is quite an important PR that needs some focus. -- This is an automated mes

Re: [PR] Use !.isEmpty() instead of .length() > 0 [iceberg]

2023-11-15 Thread via GitHub
Fokko commented on PR #9090: URL: https://github.com/apache/iceberg/pull/9090#issuecomment-1813396325 @ndrluis thanks for working on this! It looks like there are some conflicts, can you make sure that you start from the latest main branch? -- This is an automated message from the Apache

[PR] Build: Bump pydantic from 2.4.2 to 2.5.1 [iceberg-python]

2023-11-15 Thread via GitHub
dependabot[bot] opened a new pull request, #152: URL: https://github.com/apache/iceberg-python/pull/152 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.4.2 to 2.5.1. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [PR] Build: Bump pydantic from 2.4.2 to 2.5.0 [iceberg-python]

2023-11-15 Thread via GitHub
dependabot[bot] closed pull request #141: Build: Bump pydantic from 2.4.2 to 2.5.0 URL: https://github.com/apache/iceberg-python/pull/141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Build: Bump pydantic from 2.4.2 to 2.5.0 [iceberg-python]

2023-11-15 Thread via GitHub
dependabot[bot] commented on PR #141: URL: https://github.com/apache/iceberg-python/pull/141#issuecomment-1813396238 Superseded by #152. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Build: Bump rich from 13.6.0 to 13.7.0 [iceberg-python]

2023-11-15 Thread via GitHub
dependabot[bot] opened a new pull request, #151: URL: https://github.com/apache/iceberg-python/pull/151 Bumps [rich](https://github.com/Textualize/rich) from 13.6.0 to 13.7.0. Release notes Sourced from https://github.com/Textualize/rich/releases";>rich's releases. The "It's

[PR] Use !.isEmpty() instead of .length() > 0 [iceberg]

2023-11-15 Thread via GitHub
ndrluis opened a new pull request, #9090: URL: https://github.com/apache/iceberg/pull/9090 Resolves #9088 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [I] Failure to delete with merge-on-read in branch and main branch [iceberg]

2023-11-15 Thread via GitHub
amogh-jahagirdar commented on issue #7635: URL: https://github.com/apache/iceberg/issues/7635#issuecomment-1813387970 Sorry for the delay on this issue. I'm looking at it right now. Thanks for the clear reproduction steps, I can repro this and I also see this same failure happens for copy o

Re: [I] Replace `.length() > 0` with `!.isEmpty()` [iceberg]

2023-11-15 Thread via GitHub
ndrluis commented on issue #9088: URL: https://github.com/apache/iceberg/issues/9088#issuecomment-1813351839 Hello @Fokko, I am interested in resolving this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.4: Support pushing down system functions by V2 filters [iceberg]

2023-11-15 Thread via GitHub
RussellSpitzer commented on PR #7886: URL: https://github.com/apache/iceberg/pull/7886#issuecomment-1813314747 @BsoBird this is not an issue, it's a pull request. If you would like to see support for pushing down transforms into other query engine I would suggest starting a new issue for th

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-11-15 Thread via GitHub
pvary commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1813309499 @FranMorilloAWS: This issue is too light on details. Could you please try to find what happens when the data is missing? Is it a missing checkpoint? Is it a missing commit? Is there a

Re: [PR] Build: Bump duckdb from 0.9.1 to 0.9.2 [iceberg-python]

2023-11-15 Thread via GitHub
Fokko merged PR #146: URL: https://github.com/apache/iceberg-python/pull/146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mkdocstrings from 0.23.0 to 0.24.0 [iceberg-python]

2023-11-15 Thread via GitHub
Fokko merged PR #147: URL: https://github.com/apache/iceberg-python/pull/147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mypy-boto3-glue from 1.28.77 to 1.29.0 [iceberg-python]

2023-11-15 Thread via GitHub
Fokko merged PR #148: URL: https://github.com/apache/iceberg-python/pull/148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump aiohttp from 3.8.5 to 3.8.6 [iceberg-python]

2023-11-15 Thread via GitHub
Fokko merged PR #149: URL: https://github.com/apache/iceberg-python/pull/149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-11-15 Thread via GitHub
FranMorilloAWS opened a new issue, #9089: URL: https://github.com/apache/iceberg/issues/9089 ### Apache Iceberg version 0.14.1 ### Query engine Flink ### Please describe the bug 🐞 Intermittent issues with an Apache Flink Job that is consuming data from a Ki

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394744515 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394743356 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394741412 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394741412 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394737798 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Add ManifestWrite benchmark [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on code in PR #8637: URL: https://github.com/apache/iceberg/pull/8637#discussion_r1394737798 ## core/src/jmh/java/org/apache/iceberg/ManifestWriteBenchmark.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

2023-11-15 Thread via GitHub
aokolnychyi commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-1813161570 @rustyconover @Fokko, do we need any changes in readers to benefit from this? If not, can we run some existing benchmarks to showcase the read improvement is as we anticipate?

Re: [I] Iceberg: Partition-Level Tagging Support [iceberg]

2023-11-15 Thread via GitHub
Am1rr3zA commented on issue #9060: URL: https://github.com/apache/iceberg/issues/9060#issuecomment-1812951245 @nastra I might be wrong but what Apache iceberg support right now is tagging at Table level and tag is unique what I want is tagging at partition level and tag not to be unique .

Re: [PR] Support catalog name in table identifier during load, rename, drop, and purge [iceberg-python]

2023-11-15 Thread via GitHub
pdames commented on PR #150: URL: https://github.com/apache/iceberg-python/pull/150#issuecomment-1812780659 FYI @Fokko and @danielcweeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Spark: Add serialzable isolation test for concurrent MERGE INTOs [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9050: URL: https://github.com/apache/iceberg/pull/9050#discussion_r1394354418 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -390,7 +390,7 @@ private Expression conflictDetectionFilter() { filter

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9061: URL: https://github.com/apache/iceberg/pull/9061#discussion_r1394330718 ## gcp/src/main/java/org/apache/iceberg/gcp/GCPProperties.java: ## @@ -18,13 +18,15 @@ */ package org.apache.iceberg.gcp; +import com.google.api.client.util.Precond

Re: [PR] Remove `com.palantir.baseline-exact-dependencies` plugin [iceberg]

2023-11-15 Thread via GitHub
snazy commented on PR #7968: URL: https://github.com/apache/iceberg/pull/7968#issuecomment-1812685870 Yea - the plugin produces issues for us as mentioned in the description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Issue with CALL parsing [iceberg]

2023-11-15 Thread via GitHub
sundhar010 commented on issue #8343: URL: https://github.com/apache/iceberg/issues/8343#issuecomment-1812590414 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
pvary commented on PR #8553: URL: https://github.com/apache/iceberg/pull/8553#issuecomment-1812436783 @stevenzwu, @sundargates: Finally, I was able to get #8803 in. So I updated the PR, and the new versions allows to set a `Timestamp` column name as a watermark source column: ``` Ice

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1394119859 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergTimestampWatermarkExtractor.java: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1394118479 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergWatermarkExtractor.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-15 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1394117352 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -429,6 +440,20 @@ public Builder setAll(Map properties) { return this;

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
snazy commented on code in PR #9061: URL: https://github.com/apache/iceberg/pull/9061#discussion_r1394092708 ## gcp/src/main/java/org/apache/iceberg/gcp/GCPProperties.java: ## @@ -18,13 +18,15 @@ */ package org.apache.iceberg.gcp; +import com.google.api.client.util.Precondi

Re: [I] java.lang.IllegalStateException: Connection pool shut down when refreshing table metadata on s3 [iceberg]

2023-11-15 Thread via GitHub
elkhand commented on issue #8601: URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1812270095 Sure @nastra , here is the stacktrace: ``` {"exception":{ "exception_class":"org.apache.flink.util.FlinkException","exception_message":"Global failure triggered by Opera

Re: [I] java.lang.IllegalStateException: Connection pool shut down when refreshing table metadata on s3 [iceberg]

2023-11-15 Thread via GitHub
nastra commented on issue #8601: URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1812168870 @elkhand can you paste the stacktrace that you're seeing? This would help in seeing whether it's the same or a different issue. -- This is an automated message from the Apache Git S

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-15 Thread via GitHub
pvary commented on PR #8803: URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1812123613 Thanks @nastra, @rdblue, @stevenzwu, @aokolnychyi for the diligent reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] spark-procedures migrating tables can pose fatal problems [iceberg]

2023-11-15 Thread via GitHub
manuzhang commented on issue #8425: URL: https://github.com/apache/iceberg/issues/8425#issuecomment-1812121614 Our users would like to have Iceberg's table data separated from the backup data after migration. Hence, I modified the `migrate` procedure such that when a `location` for the Iceb

Re: [I] java.lang.IllegalStateException: Connection pool shut down when refreshing table metadata on s3 [iceberg]

2023-11-15 Thread via GitHub
elkhand commented on issue #8601: URL: https://github.com/apache/iceberg/issues/8601#issuecomment-1812104987 This issue still exists in the Iceberg 1.14.1 version with Flink 1.17 when the iceberg catalog is created with `'io-impl'='org.apache.iceberg.aws.s3.S3FileIO'`. The issue happens

Re: [I] spark-procedures migrating tables can pose fatal problems [iceberg]

2023-11-15 Thread via GitHub
vinnielhj commented on issue #8425: URL: https://github.com/apache/iceberg/issues/8425#issuecomment-1812067883 Sorry, I haven't followed up on this question for a long time due to personal reasons. I think the backup table can be designed as an external table, so that after the back

[I] Create JUnit5-version of FlinkTestBase [iceberg]

2023-11-15 Thread via GitHub
nastra opened a new issue, #9078: URL: https://github.com/apache/iceberg/issues/9078 ### Feature Request / Improvement `FlinkTestBase` is using JUnit4 imports/mechanisms and is used as a base class for a lot of tests. In order to be able to migrate tests to JUnit5 that depend on this

[I] Create JUnit5-version of TestFlinkScan [iceberg]

2023-11-15 Thread via GitHub
nastra opened a new issue, #9077: URL: https://github.com/apache/iceberg/issues/9077 ### Feature Request / Improvement `TestFlinkScan` is using JUnit4 imports/mechanisms and is used as a base class for a lot of tests. In order to be able to migrate tests to JUnit5 that depend on this

[I] Create JUnit5-version of SparkTestBase [iceberg]

2023-11-15 Thread via GitHub
nastra opened a new issue, #9074: URL: https://github.com/apache/iceberg/issues/9074 ### Feature Request / Improvement `SparkTestBase` is using JUnit4 imports/mechanisms and is used as a base class for a lot of tests. In order to be able to migrate tests to JUnit5 that depend on thi

[I] Create JUnit5-version of TableTestBase [iceberg]

2023-11-15 Thread via GitHub
nastra opened a new issue, #9073: URL: https://github.com/apache/iceberg/issues/9073 ### Feature Request / Improvement `TableTestBase` is using JUnit4 imports/mechanisms and is used as a base class for a lot of tests. In order to be able to migrate tests to JUnit5 that depend on this

Re: [PR] Azure: Allow shared-key auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on PR #9068: URL: https://github.com/apache/iceberg/pull/9068#issuecomment-1811986098 @bryanck could you review this one as well please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Azure: Allow shared-key auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9068: URL: https://github.com/apache/iceberg/pull/9068#discussion_r1393807727 ## azure/src/test/java/org/apache/iceberg/azure/AzurePropertiesTest.java: ## @@ -90,4 +93,34 @@ public void testNoConnectionString() { props.applyClientConfiguratio

Re: [PR] Azure: Allow shared-key auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9068: URL: https://github.com/apache/iceberg/pull/9068#discussion_r1393807167 ## azure/src/test/java/org/apache/iceberg/azure/AzurePropertiesTest.java: ## @@ -90,4 +93,34 @@ public void testNoConnectionString() { props.applyClientConfiguratio

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on PR #9061: URL: https://github.com/apache/iceberg/pull/9061#issuecomment-1811980461 @bryanck could you review this one please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9061: URL: https://github.com/apache/iceberg/pull/9061#discussion_r1393802495 ## gcp/src/main/java/org/apache/iceberg/gcp/GCPProperties.java: ## @@ -18,13 +18,15 @@ */ package org.apache.iceberg.gcp; +import com.google.api.client.util.Precond

Re: [PR] GCS: Allow no-auth for testing purposes [iceberg]

2023-11-15 Thread via GitHub
nastra commented on code in PR #9061: URL: https://github.com/apache/iceberg/pull/9061#discussion_r1393802050 ## gcp/src/test/java/org/apache/iceberg/gcp/GCPPropertiesTest.java: ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

[PR] Support catalog name in table identifier during load, rename, drop, and purge [iceberg-python]

2023-11-15 Thread via GitHub
pdames opened a new pull request, #150: URL: https://github.com/apache/iceberg-python/pull/150 **Related Issue**: https://github.com/apache/iceberg-python/issues/123 **Tests Run**: `make test` `make test-integration` **Description**: This PR fixes a bug where `Catalog`