Re: [I] Error while change log view creation using spark sql [iceberg]

2024-10-09 Thread via GitHub
Shekharrajak closed issue #11295: Error while change log view creation using spark sql URL: https://github.com/apache/iceberg/issues/11295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[I] Error while change log view creation using spark sql [iceberg]

2024-10-09 Thread via GitHub
Shekharrajak opened a new issue, #11295: URL: https://github.com/apache/iceberg/issues/11295 ### Apache Iceberg version 1.6.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Error : ``` spark-sql (nyc)> CALL demo.syst

Re: [I] Error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-10-09 Thread via GitHub
a-agmon commented on issue #655: URL: https://github.com/apache/iceberg-rust/issues/655#issuecomment-2403902431 Thanks @FANNG1 FYI @sdd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Spec: Support geo type [iceberg]

2024-10-09 Thread via GitHub
Kontinuation commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1793717159 ## format/spec.md: ## @@ -1286,6 +1291,7 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** | N

Re: [PR] API, Core: Add scan planning api models and parsers [iceberg]

2024-10-09 Thread via GitHub
amogh-jahagirdar commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1794458088 ## core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java: ## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundati

[PR] [WIP] API, Core: Proof of concept implementation of the geo support proposal [iceberg]

2024-10-09 Thread via GitHub
Kontinuation opened a new pull request, #11293: URL: https://github.com/apache/iceberg/pull/11293 This is a proof-of-concept implementation of https://github.com/apache/iceberg/pull/10981. It only includes modifications to the API, Core and Parquet modules. Modifications to the Spark module

Re: [PR] fix(arrow): Use new ParquetMetaDataReader instead [iceberg-rust]

2024-10-09 Thread via GitHub
liurenjie1024 merged PR #661: URL: https://github.com/apache/iceberg-rust/pull/661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] API, Core: Add scan planning api models and parsers [iceberg]

2024-10-09 Thread via GitHub
amogh-jahagirdar commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1794453955 ## core/src/main/java/org/apache/iceberg/rest/RESTContentFileParser.java: ## @@ -0,0 +1,250 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] API, Core: Add scan planning api models and parsers [iceberg]

2024-10-09 Thread via GitHub
amogh-jahagirdar commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1794462382 ## .palantir/revapi.yml: ## @@ -1058,6 +1058,11 @@ acceptedBreaks: new: "method void org.apache.iceberg.encryption.PlaintextEncryptionManager::()"

[PR] feat(catalog/glue): add support for list namespaces [iceberg-go]

2024-10-09 Thread via GitHub
oguzerdogmus opened a new pull request, #169: URL: https://github.com/apache/iceberg-go/pull/169 Add support for `ListNamespaces` to the Glue catalog. ### Implementation Details - Similar to tables, used the same `ICEBERG` parameter to differentiate between Iceberg catalog data

Re: [PR] API, Core: Add scan planning api models and parsers [iceberg]

2024-10-09 Thread via GitHub
amogh-jahagirdar commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1794452438 ## core/src/main/java/org/apache/iceberg/GenericDataFile.java: ## @@ -26,7 +26,7 @@ import org.apache.iceberg.relocated.com.google.common.collect.ImmutableM

Re: [PR] Spark: Merge new position deletes with old deletes during writing [iceberg]

2024-10-09 Thread via GitHub
singhpk234 commented on code in PR #11273: URL: https://github.com/apache/iceberg/pull/11273#discussion_r1794359228 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java: ## @@ -158,6 +163,26 @@ public void filter(Predicate[] predicates) {

Re: [PR] Spark: Merge new position deletes with old deletes during writing [iceberg]

2024-10-09 Thread via GitHub
singhpk234 commented on code in PR #11273: URL: https://github.com/apache/iceberg/pull/11273#discussion_r1794359228 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java: ## @@ -158,6 +163,26 @@ public void filter(Predicate[] predicates) {

Re: [I] Error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-10-09 Thread via GitHub
FANNG1 commented on issue #655: URL: https://github.com/apache/iceberg-rust/issues/655#issuecomment-2403722581 This is mainly caused by https://github.com/apache/datafusion/issues/12687, I will propose a PR to upgrade datafusion version. -- This is an automated message from the Apache Gi

Re: [PR] Remove spring-boot dependency [iceberg]

2024-10-09 Thread via GitHub
manuzhang commented on code in PR #11291: URL: https://github.com/apache/iceberg/pull/11291#discussion_r1794423421 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -121,8 +116,9 @@ void deleteBucket(String bucketName) throws IOExcept

Re: [I] Request Timeout API to RestCatalog's HTTPClient is provided by Iceberg SDK [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8915: Request Timeout API to RestCatalog's HTTPClient is provided by Iceberg SDK URL: https://github.com/apache/iceberg/issues/8915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Schema issue between Arrow and PyIceberg [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8913: URL: https://github.com/apache/iceberg/issues/8913#issuecomment-2403653796 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Request Timeout API to RestCatalog's HTTPClient is provided by Iceberg SDK [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8915: URL: https://github.com/apache/iceberg/issues/8915#issuecomment-2403653811 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Pushdown SUBSTRING filter when equivalent to STARTSWITH [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8911: Pushdown SUBSTRING filter when equivalent to STARTSWITH URL: https://github.com/apache/iceberg/issues/8911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Consumer Latency Monitoring Support in Iceberg ? [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8903: URL: https://github.com/apache/iceberg/issues/8903#issuecomment-2403653743 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Pushdown SUBSTRING filter when equivalent to STARTSWITH [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8911: URL: https://github.com/apache/iceberg/issues/8911#issuecomment-2403653776 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Schema issue between Arrow and PyIceberg [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8913: Schema issue between Arrow and PyIceberg URL: https://github.com/apache/iceberg/issues/8913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] operations fail after upgrading to spark 3.4 [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8904: URL: https://github.com/apache/iceberg/issues/8904#issuecomment-2403653759 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] operations fail after upgrading to spark 3.4 [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8904: operations fail after upgrading to spark 3.4 URL: https://github.com/apache/iceberg/issues/8904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Consumer Latency Monitoring Support in Iceberg ? [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8903: Consumer Latency Monitoring Support in Iceberg ? URL: https://github.com/apache/iceberg/issues/8903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Hive's performance for querying the Iceberg table is very poor. [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8901: Hive's performance for querying the Iceberg table is very poor. URL: https://github.com/apache/iceberg/issues/8901 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Apache hive 3 with Tez engine select table no empty [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8891: Apache hive 3 with Tez engine select table no empty URL: https://github.com/apache/iceberg/issues/8891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Flink: OverflowError: value too large to convert to int32_t [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8874: URL: https://github.com/apache/iceberg/issues/8874#issuecomment-2403653676 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2403653646 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] fast_forward command not merging branches within AWS Glue [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8881: URL: https://github.com/apache/iceberg/issues/8881#issuecomment-2403653687 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Apache hive 3 with Tez engine select table no empty [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on issue #8891: URL: https://github.com/apache/iceberg/issues/8891#issuecomment-2403653700 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] fast_forward command not merging branches within AWS Glue [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8881: fast_forward command not merging branches within AWS Glue URL: https://github.com/apache/iceberg/issues/8881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Flink: OverflowError: value too large to convert to int32_t [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] closed issue #8874: Flink: OverflowError: value too large to convert to int32_t URL: https://github.com/apache/iceberg/issues/8874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: Add Catalog Transactions API [iceberg]

2024-10-09 Thread via GitHub
github-actions[bot] commented on PR #6948: URL: https://github.com/apache/iceberg/pull/6948#issuecomment-2403653588 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Arrow: Remove unused fixed width binary reader classes [iceberg]

2024-10-09 Thread via GitHub
wypoon commented on PR #11292: URL: https://github.com/apache/iceberg/pull/11292#issuecomment-2403628522 @nastra I missed noticing earlier that the fixed width binary reader classes are no longer used. I think this is it, though. -- This is an automated message from the Apache Git Serv

Re: [PR] OpenAPI: Add planning-mode to loadTable response [iceberg]

2024-10-09 Thread via GitHub
rahil-c commented on PR #11156: URL: https://github.com/apache/iceberg/pull/11156#issuecomment-2403622257 Was wondering if I could get your thoughts on this @rdblue @danielcweeks @amogh-jahagirdar @singhpk234 ? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] API, Core: Add scan planning api request and response models [iceberg]

2024-10-09 Thread via GitHub
rahil-c commented on PR #11180: URL: https://github.com/apache/iceberg/pull/11180#issuecomment-2403620243 @rdblue @danielcweeks @amogh-jahagirdar @jackye1995 @singhpk234 When implementing the parsers, one thing that I noticed was that not having a `partition-spec` sent back from the servic

Re: [PR] Arrow: Remove unused fixed width binary reader classes [iceberg]

2024-10-09 Thread via GitHub
wypoon commented on code in PR #11292: URL: https://github.com/apache/iceberg/pull/11292#discussion_r1794387844 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -278,10 +264,6 @@ public VarWidthTypeBatchReader varWidthTypeBa

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794384589 ## format/puffin-spec.md: ## @@ -123,6 +123,49 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794383706 ## spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java: ## @@ -108,14 +108,14 @@ public static Object[][] parameters() {

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1794381271 ## format/spec.md: ## @@ -454,35 +457,40 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1794381735 ## format/spec.md: ## @@ -454,35 +457,40 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Spark: Merge new position deletes with old deletes during writing [iceberg]

2024-10-09 Thread via GitHub
singhpk234 commented on code in PR #11273: URL: https://github.com/apache/iceberg/pull/11273#discussion_r1794359228 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java: ## @@ -158,6 +163,26 @@ public void filter(Predicate[] predicates) {

Re: [PR] Spark: Merge new position deletes with old deletes during writing [iceberg]

2024-10-09 Thread via GitHub
singhpk234 commented on code in PR #11273: URL: https://github.com/apache/iceberg/pull/11273#discussion_r1794359228 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java: ## @@ -158,6 +163,26 @@ public void filter(Predicate[] predicates) {

Re: [PR] Spark: Merge new position deletes with old deletes during writing [iceberg]

2024-10-09 Thread via GitHub
singhpk234 commented on code in PR #11273: URL: https://github.com/apache/iceberg/pull/11273#discussion_r1794316342 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -383,4 +383,8 @@ private TableProperties() {} public static final int ENCRYPTION_DEK_LENGTH

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1794325498 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends VectorHolder

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1794325498 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends VectorHolder

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1794325498 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends VectorHolder

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on PR #11257: URL: https://github.com/apache/iceberg/pull/11257#issuecomment-2403559725 @RussellSpitzer Thanks for your review! I have addressed the comments and switched back to Preview1, along with reverting a few changes I made for Preview2/snapshot. I switched back t

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1794325342 ## format/spec.md: ## @@ -841,14 +842,38 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] Bump getdaft from 0.3.2 to 0.3.5 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] closed pull request #1214: Bump getdaft from 0.3.2 to 0.3.5 URL: https://github.com/apache/iceberg-python/pull/1214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] Bump getdaft from 0.3.2 to 0.3.6 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] opened a new pull request, #1225: URL: https://github.com/apache/iceberg-python/pull/1225 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.3.2 to 0.3.6. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [PR] Bump getdaft from 0.3.2 to 0.3.5 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] commented on PR #1214: URL: https://github.com/apache/iceberg-python/pull/1214#issuecomment-2403555295 Superseded by #1225. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794322243 ## spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java: ## @@ -108,14 +108,14 @@ public static Object[][] parameters() {

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794322082 ## spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -138,6 +138,7 @@ public class TestRewriteDataFilesAction e

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794321900 ## spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownInRowLevelOperations.java: ## @@ -260,7 +267,12 @@ privat

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794321692 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala: ## @@ -30,7 +30,7 @@ import org.

Re: [PR] Initial Support for Spark 4.0 preview [iceberg]

2024-10-09 Thread via GitHub
huaxingao commented on code in PR #11257: URL: https://github.com/apache/iceberg/pull/11257#discussion_r1794321464 ## gradle/libs.versions.toml: ## @@ -47,6 +47,7 @@ flink120 = { strictly = "1.20.0"} google-libraries-bom = "26.47.0" guava = "33.3.0-jre" hadoop2 = "2.7.3" +had

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1794307274 ## format/spec.md: ## @@ -841,14 +842,38 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] Bump pypa/cibuildwheel from 2.21.1 to 2.21.2 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] commented on PR #1216: URL: https://github.com/apache/iceberg-python/pull/1216#issuecomment-2403520507 Superseded by #1224. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Bump pypa/cibuildwheel from 2.21.1 to 2.21.2 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] closed pull request #1216: Bump pypa/cibuildwheel from 2.21.1 to 2.21.2 URL: https://github.com/apache/iceberg-python/pull/1216 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[PR] Bump pypa/cibuildwheel from 2.21.1 to 2.21.3 [iceberg-python]

2024-10-09 Thread via GitHub
dependabot[bot] opened a new pull request, #1224: URL: https://github.com/apache/iceberg-python/pull/1224 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.21.1 to 2.21.3. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwh

Re: [PR] Flink: Tests alignment for the Flink Sink v2-based implemenation (IcebergSink) [iceberg]

2024-10-09 Thread via GitHub
rodmeneses commented on PR #11219: URL: https://github.com/apache/iceberg/pull/11219#issuecomment-2403492186 Hi @arkadius I have started working in backporting the RANGE distribution to the IcebergSink. The unit tests in my code will benefit from the new marker interface you are introdu

Re: [PR] Flink: FlinkSink & IcebergSink desynchronized tests alignment [iceberg]

2024-10-09 Thread via GitHub
rodmeneses commented on PR #11249: URL: https://github.com/apache/iceberg/pull/11249#issuecomment-2403483118 @pvary @stevenzwu could you guys please start the CI pipelines on this PR? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794250448 ## format/puffin-spec.md: ## @@ -123,6 +123,49 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct

Re: [PR] Spec: Add v3 types and type promotion [iceberg]

2024-10-09 Thread via GitHub
rdblue merged PR #10955: URL: https://github.com/apache/iceberg/pull/10955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794238145 ## format/puffin-spec.md: ## @@ -123,6 +123,44 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

Re: [PR] [For testing only] Testing the BaseIncrementalChangelogScan implementation from #9888 [iceberg]

2024-10-09 Thread via GitHub
wypoon closed pull request #10954: [For testing only] Testing the BaseIncrementalChangelogScan implementation from #9888 URL: https://github.com/apache/iceberg/pull/10954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794227286 ## format/puffin-spec.md: ## @@ -123,6 +123,44 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794214522 ## format/puffin-spec.md: ## @@ -123,6 +123,44 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794213504 ## format/puffin-spec.md: ## @@ -123,6 +123,44 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

Re: [PR] More accurate estimate on parquet row groups size [iceberg]

2024-10-09 Thread via GitHub
jinyangli34 commented on PR #11258: URL: https://github.com/apache/iceberg/pull/11258#issuecomment-2403412692 Run benchmark again, increased `NUM_RECORDS` from 1M to 5M Tested 4 groups: **main**: main branch without change in this PR **PR**: this PR **PR+2**: two more getBuffe

Re: [PR] Puffin: Add delete-vector-v1 blob type [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #11238: URL: https://github.com/apache/iceberg/pull/11238#discussion_r1794211865 ## format/puffin-spec.md: ## @@ -123,6 +123,44 @@ The blob metadata for this blob may include following properties: - `ndv`: estimate of number of distinct values,

[PR] Remove spring-boot dependency [iceberg]

2024-10-09 Thread via GitHub
jbonofre opened a new pull request, #11291: URL: https://github.com/apache/iceberg/pull/11291 This PR removes the spring-boot dependency from Iceberg and implement Aliyun OSS Mock using the JDK HTTP server. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Spec: Add v3 types and type promotion [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #10955: URL: https://github.com/apache/iceberg/pull/10955#discussion_r1794205253 ## format/spec.md: ## @@ -1089,6 +1118,7 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | Primitive type | Hash specificat

Re: [PR] Spec: Add v3 types and type promotion [iceberg]

2024-10-09 Thread via GitHub
rdblue commented on code in PR #10955: URL: https://github.com/apache/iceberg/pull/10955#discussion_r1794202328 ## format/spec.md: ## @@ -1089,6 +1118,7 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | Primitive type | Hash specificat

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1794135680 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns i

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1794135310 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns i

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1794079437 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns i

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1794072935 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends Vector

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1794025905 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends Vector

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
sumedhsakdeo commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1794016589 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
RussellSpitzer commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1794003825 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns i

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-09 Thread via GitHub
sumedhsakdeo commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1793865228 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [I] Support commit retries [iceberg-python]

2024-10-09 Thread via GitHub
maxlucuta commented on issue #269: URL: https://github.com/apache/iceberg-python/issues/269#issuecomment-2402969232 Have also have experience not being able write to tables in highly distributed environments. Refreshing the table in isolation, in addition to adding some retry logic did not

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-09 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1793922257 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -262,6 +265,142 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] open-api: Build runtime jar for test fixture [iceberg]

2024-10-09 Thread via GitHub
danielcweeks commented on code in PR #11279: URL: https://github.com/apache/iceberg/pull/11279#discussion_r1793937666 ## build.gradle: ## @@ -1006,6 +1009,37 @@ project(':iceberg-open-api') { recommend.set(true) } check.dependsOn('validateRESTCatalogSpec') + + // Cre

Re: [I] Support commit retries [iceberg-python]

2024-10-09 Thread via GitHub
kevinjqliu commented on issue #269: URL: https://github.com/apache/iceberg-python/issues/269#issuecomment-2402909492 As a workaround, to manually retry commits, update the table metadata by using ``` table = table.refresh() ``` before calling `commit()` again -- This is an

[PR] feat(table): Implement converting Iceberg schema and types to Arrow [iceberg-go]

2024-10-09 Thread via GitHub
zeroshade opened a new pull request, #168: URL: https://github.com/apache/iceberg-go/pull/168 #155 implemented the conversion of Arrow schemas to Iceberg which will be needed for reading data from Parquet files or otherwise. This PR implements the reverse, converting Iceberg schemas and typ

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-10-09 Thread via GitHub
danielcweeks commented on PR #2: URL: https://github.com/apache/iceberg/pull/2#issuecomment-2402888017 > What do you think about this approach @danielcweeks: > > > Is there an optimal number of directories and depth? maybe we can just create those and put rest of the entropy i

Re: [PR] OpenAPI: Define REST Catalog models for Row-Level Updates [iceberg]

2024-10-09 Thread via GitHub
jackye1995 commented on code in PR #11287: URL: https://github.com/apache/iceberg/pull/11287#discussion_r1793872507 ## open-api/rest-catalog-open-api.yaml: ## @@ -3082,6 +3132,47 @@ components: default-sort-order-id: type: integer +AssertOverwriteRows:

Re: [PR] OpenAPI: Define REST Catalog models for Row-Level Updates [iceberg]

2024-10-09 Thread via GitHub
geruh commented on code in PR #11287: URL: https://github.com/apache/iceberg/pull/11287#discussion_r1793870830 ## open-api/rest-catalog-open-api.py: ## @@ -896,19 +896,6 @@ class SetPartitionStatisticsUpdate(BaseUpdate): ) -class TableRequirement(BaseModel): Review Com

Re: [PR] OpenAPI: Define REST Catalog models for Row-Level Updates [iceberg]

2024-10-09 Thread via GitHub
jackye1995 commented on code in PR #11287: URL: https://github.com/apache/iceberg/pull/11287#discussion_r1793868108 ## open-api/rest-catalog-open-api.py: ## @@ -896,19 +896,6 @@ class SetPartitionStatisticsUpdate(BaseUpdate): ) -class TableRequirement(BaseModel): Revie

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.41 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] commented on PR #162: URL: https://github.com/apache/iceberg-go/pull/162#issuecomment-2402837888 Superseded by #167. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.61.2 to 1.65.0 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] commented on PR #164: URL: https://github.com/apache/iceberg-go/pull/164#issuecomment-2402837310 Superseded by #166. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.61.2 to 1.65.0 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] closed pull request #164: build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.61.2 to 1.65.0 URL: https://github.com/apache/iceberg-go/pull/164 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.37 to 1.17.39 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] commented on PR #161: URL: https://github.com/apache/iceberg-go/pull/161#issuecomment-2402837089 Superseded by #165. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.43 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] opened a new pull request, #167: URL: https://github.com/apache/iceberg-go/pull/167 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.39 to 1.27.43. Commits https://github.com/aws/aws-sdk-go-v2/commit/0cbb5aa17f9078cb45

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.61.2 to 1.65.2 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] opened a new pull request, #166: URL: https://github.com/apache/iceberg-go/pull/166 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.61.2 to 1.65.2. Commits https://github.com/aws/aws-sdk-go-v2/commit/0cbb5aa17f9078cb

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.41 [iceberg-go]

2024-10-09 Thread via GitHub
dependabot[bot] closed pull request #162: build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.41 URL: https://github.com/apache/iceberg-go/pull/162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

  1   2   >