Re: [I] Spark returns incorrect results when reading Parquet bloom filters created by Trino [iceberg]

2025-04-05 Thread via GitHub
ebyhr commented on issue #12458: URL: https://github.com/apache/iceberg/issues/12458#issuecomment-2781253500 @hsiang-c This is reproducible with Trino 474. Can you try Spark 3.4.2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Upsert with list type not supported [iceberg-python]

2025-04-05 Thread via GitHub
codrut20 commented on issue #1711: URL: https://github.com/apache/iceberg-python/issues/1711#issuecomment-2781248484 will this be part of 0.9.1 release? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Metadata `entries` table breaks when the table configured as Merge-on-Read and has Delete Files [iceberg-python]

2025-04-05 Thread via GitHub
guptaakashdeep commented on issue #1884: URL: https://github.com/apache/iceberg-python/issues/1884#issuecomment-2781236612 @kevinjqliu Please let me know if we need to add more details that needs to be added in here. I further looked into the code to fix the issue and it seems to be

[I] Metadata `entries` table breaks when the table configured as Merge-on-Read and has Delete Files [iceberg-python]

2025-04-05 Thread via GitHub
guptaakashdeep opened a new issue, #1884: URL: https://github.com/apache/iceberg-python/issues/1884 ### Apache Iceberg version 0.9.0 (latest release) ### Please describe the bug ๐Ÿž ## Issue: `table.inspect.entries()` fails when table is MOR table and has Delete Files p

Re: [D] Glue catalog updating [iceberg-rust]

2025-04-05 Thread via GitHub
GitHub user hugokitano edited a discussion: Glue catalog updating I'm able to write parquet and metadata/manifest files with the **Glue** catalog, but I am not able to see the catalog update with snapshots, and writing subsequently to the same table makes it clear the catalog always thinks it

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.52 to 2.31.11 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] commented on PR #12690: URL: https://github.com/apache/iceberg/pull/12690#issuecomment-2781220711 Superseded by #12734. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.52 to 2.31.11 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] closed pull request #12690: Build: Bump software.amazon.awssdk:bom from 2.29.52 to 2.31.11 URL: https://github.com/apache/iceberg/pull/12690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump software.amazon.awssdk:bom from 2.29.52 to 2.31.16 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12734: URL: https://github.com/apache/iceberg/pull/12734 Bumps software.amazon.awssdk:bom from 2.29.52 to 2.31.16. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.1 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] closed pull request #12535: Build: Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.1 URL: https://github.com/apache/iceberg/pull/12535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump com.google.cloud:libraries-bom from 26.58.0 to 26.59.0 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12733: URL: https://github.com/apache/iceberg/pull/12733 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.58.0 to 26.59.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bo

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.2 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12732: URL: https://github.com/apache/iceberg/pull/12732 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.23.0 to 3.23.2. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/

[PR] Build: Bump io.delta:delta-standalone_2.12 from 3.3.0 to 3.3.1 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12731: URL: https://github.com/apache/iceberg/pull/12731 Bumps [io.delta:delta-standalone_2.12](https://github.com/delta-io/delta) from 3.3.0 to 3.3.1. Commits https://github.com/delta-io/delta/commit/bee74a2cfd282875f8c08f6f54c76fad4

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.1 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] commented on PR #12535: URL: https://github.com/apache/iceberg/pull/12535#issuecomment-2781220614 Superseded by #12732. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] Build: Bump io.netty:netty-buffer from 4.1.119.Final to 4.2.0.Final [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12730: URL: https://github.com/apache/iceberg/pull/12730 Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.1.119.Final to 4.2.0.Final. Commits https://github.com/netty/netty/commit/09e64d259c99be8b5b2a471a78f11e65eb

[PR] Build: Bump io.delta:delta-spark_2.12 from 3.3.0 to 3.3.1 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12729: URL: https://github.com/apache/iceberg/pull/12729 Bumps [io.delta:delta-spark_2.12](https://github.com/delta-io/delta) from 3.3.0 to 3.3.1. Commits https://github.com/delta-io/delta/commit/bee74a2cfd282875f8c08f6f54c76fad49afa5

[PR] Build: Bump mkdocs-material from 9.6.9 to 9.6.11 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #12728: URL: https://github.com/apache/iceberg/pull/12728 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.9 to 9.6.11. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs

[I] Glue Catalog updating [iceberg-rust]

2025-04-05 Thread via GitHub
hugokitano opened a new issue, #1169: URL: https://github.com/apache/iceberg-rust/issues/1169 ### Is your feature request related to a problem or challenge? I'm able to write parquet and metadata/manifest files with the **Glue** catalog, but I am not able to see the catalog update wit

Re: [PR] List data and metadata directories instead of table root [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] commented on PR #12278: URL: https://github.com/apache/iceberg/pull/12278#issuecomment-2781144999 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Avoid Avro recursive schema for Variant schema. [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] commented on PR #12459: URL: https://github.com/apache/iceberg/pull/12459#issuecomment-2781145015 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatโ€™s incorrect or this pul

Re: [PR] Fix versions in LICENSE/NOTICE [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] closed pull request #12364: Fix versions in LICENSE/NOTICE URL: https://github.com/apache/iceberg/pull/12364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] List data and metadata directories instead of table root [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] closed pull request #12278: List data and metadata directories instead of table root URL: https://github.com/apache/iceberg/pull/12278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Fix versions in LICENSE/NOTICE [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] commented on PR #12364: URL: https://github.com/apache/iceberg/pull/12364#issuecomment-2781145003 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] How to avoid partition key sorting when inserting data into a partitioned Iceberg table? [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] commented on issue #10181: URL: https://github.com/apache/iceberg/issues/10181#issuecomment-2781144966 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] handle decimal physicial type mapping [iceberg-python]

2025-04-05 Thread via GitHub
redpheonixx commented on code in PR #1799: URL: https://github.com/apache/iceberg-python/pull/1799#discussion_r2001293953 ## tests/io/test_pyarrow_stats.py: ## @@ -72,7 +72,7 @@ StringType, ) from pyiceberg.utils.datetime import date_to_days, datetime_to_micros, time_to_

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-05 Thread via GitHub
ajantha-bhat commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2016876100 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -135,20 +142,114 @@ public static PartitionStatisticsFile computeAndWriteStatsFi

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-05 Thread via GitHub
pvary commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2010120536 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] ViewVersionLog::timestamp consumes self [iceberg-rust]

2025-04-05 Thread via GitHub
Xuanwo closed issue #1100: ViewVersionLog::timestamp consumes self URL: https://github.com/apache/iceberg-rust/issues/1100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-05 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021373556 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] Docs: Fix lifecycle and versions in multi-engine-support [iceberg]

2025-04-05 Thread via GitHub
nastra merged PR #12370: URL: https://github.com/apache/iceberg/pull/12370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core, Parquet, ORC: Fix missing data when writing unknown [iceberg]

2025-04-05 Thread via GitHub
rdblue commented on code in PR #12581: URL: https://github.com/apache/iceberg/pull/12581#discussion_r2005949897 ## orc/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java: ## @@ -156,8 +156,8 @@ public Stream> metrics() { private static class RecordWriter extend

Re: [PR] feat: add partition field/partition spec [iceberg-cpp]

2025-04-05 Thread via GitHub
gty404 commented on code in PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#discussion_r2022528676 ## src/iceberg/transform.h: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the

[PR] chore(deps): Bump arrow-buffer from 54.2.0 to 54.3.0 [iceberg-rust]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1127: URL: https://github.com/apache/iceberg-rust/pull/1127 Bumps [arrow-buffer](https://github.com/apache/arrow-rs) from 54.2.0 to 54.3.0. Release notes Sourced from https://github.com/apache/arrow-rs/releases";>arrow-buffer's releases.

Re: [PR] Core: Support first-row-id for manifests and manifest lists [iceberg]

2025-04-05 Thread via GitHub
rdblue commented on code in PR #12672: URL: https://github.com/apache/iceberg/pull/12672#discussion_r2019438377 ## core/src/main/java/org/apache/iceberg/V3Metadata.java: ## @@ -140,6 +143,22 @@ private Object get(int pos) { return wrapped.partitions(); case 1

Re: [PR] Fix decimal physicial type mapping [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on PR #1839: URL: https://github.com/apache/iceberg-python/pull/1839#issuecomment-2766266153 Thanks for fixing this @redpheonixx ๐Ÿ™Œ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: Cleanup unit tests [iceberg]

2025-04-05 Thread via GitHub
sullis commented on PR #12666: URL: https://github.com/apache/iceberg/pull/12666#issuecomment-2761846481 I pushed an update to this branch. I identified 34 unit tests that needed cleanup. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] feat: re-export name mapping [iceberg-rust]

2025-04-05 Thread via GitHub
liurenjie1024 commented on PR #1116: URL: https://github.com/apache/iceberg-rust/pull/1116#issuecomment-2742142290 > @jdockerty @liurenjie1024 I believe #1072 contains a lot of the functionality in this pr, this got split into #1082 being the first part of it. Hi, @jonathanc-n I think

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-05 Thread via GitHub
zratkai commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2005885458 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java: ## @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[I] Support hms catalog in cli tool [iceberg-rust]

2025-04-05 Thread via GitHub
liurenjie1024 opened a new issue, #1157: URL: https://github.com/apache/iceberg-rust/issues/1157 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] add scan tests with null values [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on code in PR #1865: URL: https://github.com/apache/iceberg-python/pull/1865#discussion_r2021670570 ## tests/io/test_pyarrow.py: ## @@ -2317,3 +2321,66 @@ def test_pyarrow_io_multi_fs() -> None: # Same PyArrowFileIO instance resolves local file input t

Re: [I] Ingestion using Iceberg bucketing causing OOM [iceberg]

2025-04-05 Thread via GitHub
RussellSpitzer commented on issue #11393: URL: https://github.com/apache/iceberg/issues/11393#issuecomment-2751788380 You are setting driver heap size a bit too late. If you want to change the driver JVM size (which you must do for local mode) you need to do it in the JVM arguements of the

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-05 Thread via GitHub
zratkai commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2009929329 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Core: lazy init workerPool [iceberg]

2025-04-05 Thread via GitHub
abstractdog commented on code in PR #12427: URL: https://github.com/apache/iceberg/pull/12427#discussion_r2026705782 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1772,6 +1775,24 @@ public void testNoSchemasOrSpecsToRemove() {

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-05 Thread via GitHub
lidavidm commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2025773673 ## src/iceberg/expression.cc: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] Rename `pyiceberg_core` to `pyiceberg-core` [iceberg-rust]

2025-04-05 Thread via GitHub
Xuanwo merged PR #1134: URL: https://github.com/apache/iceberg-rust/pull/1134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark: Add some tests for variant fixup [iceberg]

2025-04-05 Thread via GitHub
rdblue commented on code in PR #12497: URL: https://github.com/apache/iceberg/pull/12497#discussion_r2004434232 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkFixupTypes.java: ## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] CORE: Allow HTTPClient to parse headers from properties. [iceberg]

2025-04-05 Thread via GitHub
nastra commented on code in PR #12595: URL: https://github.com/apache/iceberg/pull/12595#discussion_r2012437689 ## core/src/main/java/org/apache/iceberg/rest/RESTCatalog.java: ## @@ -55,7 +55,11 @@ public class RESTCatalog public RESTCatalog() { this( SessionCat

Re: [PR] AWS: Update the aws-bundle with latest dependencies [iceberg]

2025-04-05 Thread via GitHub
ajantha-bhat commented on code in PR #12553: URL: https://github.com/apache/iceberg/pull/12553#discussion_r2013613026 ## aws-bundle/LICENSE: ## @@ -443,6 +443,12 @@ License: Apache License, Version 2.0 - https://aws.amazon.com/apache2.0 -

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-05 Thread via GitHub
deniskuzZ commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2016249058 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -45,22 +49,53 @@ private PartitionStatsUtil() {} * @param table the table for which part

[PR] Test with Iceberg-Rust [iceberg-python]

2025-04-05 Thread via GitHub
Fokko opened a new pull request, #1833: URL: https://github.com/apache/iceberg-python/pull/1833 # Rationale for this change Testing out to use Iceberg Rust for all of the transforms. I think we have some rounding error in https://github.com/apache/iceberg-rust/pull/1128/ # Are

Re: [PR] Spark: when doing rewrite_data_files, check for partitioning schema compatibility [iceberg]

2025-04-05 Thread via GitHub
adrians commented on code in PR #12651: URL: https://github.com/apache/iceberg/pull/12651#discussion_r2026447286 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -265,6 +265,22 @@ public boolean equals(Object other) { return Arrays.equals(fields, that.field

Re: [PR] Flink: backport support create table like in flink catalog [iceberg]

2025-04-05 Thread via GitHub
pvary commented on PR #12679: URL: https://github.com/apache/iceberg/pull/12679#issuecomment-2763227444 @swapna267: is this a clean backport, or you had to do changes compared to the 1.20 PR? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] decimal physicial type mapping [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on code in PR #1839: URL: https://github.com/apache/iceberg-python/pull/1839#discussion_r2021041283 ## pyiceberg/io/pyarrow.py: ## @@ -2350,8 +2351,19 @@ def data_file_statistics_from_parquet_metadata( stats_col.iceberg_type, statisti

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-04-05 Thread via GitHub
kevinjqliu commented on code in PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#discussion_r2021303617 ## tests/table/test_upsert.py: ## @@ -509,3 +509,39 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: ValueError, match="Join

Re: [PR] Spec: update to reflect lineage is required [iceberg]

2025-04-05 Thread via GitHub
danielcweeks commented on code in PR #12580: URL: https://github.com/apache/iceberg/pull/12580#discussion_r2021573693 ## format/spec.md: ## @@ -458,11 +457,11 @@ The snapshot then populates the total number of `added-rows` based on the sum of When the new snapshot is committed

Re: [I] [feature request] Support Time64Type[ns] [iceberg-python]

2025-04-05 Thread via GitHub
github-actions[bot] commented on issue #1169: URL: https://github.com/apache/iceberg-python/issues/1169#issuecomment-2767704299 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] RestCatalog append table is slow (2+s) [iceberg-python]

2025-04-05 Thread via GitHub
HungYangChang commented on issue #1806: URL: https://github.com/apache/iceberg-python/issues/1806#issuecomment-2734350396 I did some dirty logging in pyiceberg.table.append ``` def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT) -> None: """

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-04-05 Thread via GitHub
pvary closed pull request #11826: backport #11301(rowconverter) to Flink 1.19 and 1.18 URL: https://github.com/apache/iceberg/pull/11826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Migrate Spark 3.4 TestBase-related remaining tests in actions [iceberg]

2025-04-05 Thread via GitHub
nastra merged PR #12579: URL: https://github.com/apache/iceberg/pull/12579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] V3: Introduce `timestamp_ns` and `timestamptz_ns` [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on code in PR #1632: URL: https://github.com/apache/iceberg-python/pull/1632#discussion_r2009202742 ## pyiceberg/types.py: ## @@ -62,6 +63,12 @@ FIXED_PARSER = ParseNumberFromBrackets(FIXED) +class TableVersion(IntEnum): +ONE = 1 +TWO = 2 +THREE

Re: [PR] V3: Introduce `timestamp_ns` and `timestamptz_ns` [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on PR #1632: URL: https://github.com/apache/iceberg-python/pull/1632#issuecomment-2736444700 Looks like there is some issue with the `UnknownType`, we need to pass in the version there for the create table: ``` tbl = catalog.create_table( ident

Re: [PR] Core: Pass storage credentials from LoadTableResponse to FileIO [iceberg]

2025-04-05 Thread via GitHub
danielcweeks commented on code in PR #12591: URL: https://github.com/apache/iceberg/pull/12591#discussion_r2021685856 ## core/src/main/java/org/apache/iceberg/rest/responses/LoadTableResponse.java: ## @@ -80,7 +81,24 @@ public TableMetadata tableMetadata() { } public Map

Re: [PR] Doc: Add Hive 2.x/3.x support notes in hive.md [iceberg]

2025-04-05 Thread via GitHub
deniskuzZ commented on code in PR #12700: URL: https://github.com/apache/iceberg/pull/12700#discussion_r2028238705 ## docs/docs/hive.md: ## @@ -126,9 +90,6 @@ To enable Hive support globally for an application, set `iceberg.engine.hive.ena For example, setting this in the `hiv

Re: [PR] AWS: Update the aws-bundle with latest dependencies [iceberg]

2025-04-05 Thread via GitHub
SanjayMarreddi commented on code in PR #12553: URL: https://github.com/apache/iceberg/pull/12553#discussion_r2016797737 ## aws-bundle/build.gradle: ## @@ -36,6 +38,9 @@ project(":iceberg-aws-bundle") { implementation "software.amazon.awssdk:sts" implementation "softwar

Re: [PR] Core: FileRewritePlanner implementation [iceberg]

2025-04-05 Thread via GitHub
stevenzwu commented on code in PR #12493: URL: https://github.com/apache/iceberg/pull/12493#discussion_r2014688814 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,310 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core, Hive: Double check commit status in case of commit conflict for NoLock [iceberg]

2025-04-05 Thread via GitHub
deniskuzZ commented on code in PR #12637: URL: https://github.com/apache/iceberg/pull/12637#discussion_r2020560062 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -287,15 +277,33 @@ protected void doCommit(TableMetadata base, TableMetadata

Re: [I] Iceberg Read is not working on Iceberg Hive table [iceberg]

2025-04-05 Thread via GitHub
hrvylein commented on issue #11168: URL: https://github.com/apache/iceberg/issues/11168#issuecomment-2764721321 anyone figured out why this is happening? i have the same problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Implement MergeFiles operation [iceberg-go]

2025-04-05 Thread via GitHub
arnaudbriche opened a new pull request, #354: URL: https://github.com/apache/iceberg-go/pull/354 This is the first working prototype of the MergeFiles operation discussed here: https://github.com/apache/iceberg-go/issues/348 I can see that on datafile file deletion, a new manifest wit

[PR] Core: Use credentials from LoadTableResponse if available [iceberg]

2025-04-05 Thread via GitHub
nastra opened a new pull request, #12591: URL: https://github.com/apache/iceberg/pull/12591 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Spark 3.5: Support case sensitive in replace where statement [iceberg]

2025-04-05 Thread via GitHub
nastra commented on code in PR #12706: URL: https://github.com/apache/iceberg/pull/12706#discussion_r2026809626 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -351,6 +353,7 @@ private OverwriteByFilter(Expression overwriteExpr) { pu

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-04-05 Thread via GitHub
Fokko commented on PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#issuecomment-2766754471 > now im wondering if nulls are properly handled when we convert iceberg expressions to pyarrow expressions I think we need to check the `Or` and `And` `BooleanExpressions`, be

[PR] Build: Bump pydantic from 2.10.6 to 2.11.1 [iceberg-python]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1869: URL: https://github.com/apache/iceberg-python/pull/1869 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.10.6 to 2.11.1. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-05 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021373556 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] Implement MergeFiles operation [iceberg-go]

2025-04-05 Thread via GitHub
arnaudbriche commented on PR #354: URL: https://github.com/apache/iceberg-go/pull/354#issuecomment-2742904433 The implementation you worked is clearly taking a lot more things into (properties, partitioning, ...) account and I think this feature should be based on this work. Why not you

[PR] ORC: Support timestamp(9), variant, and unknown in generics [iceberg]

2025-04-05 Thread via GitHub
rdblue opened a new pull request, #12567: URL: https://github.com/apache/iceberg/pull/12567 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

[PR] Core: Add update event for rewrite manifests [iceberg]

2025-04-05 Thread via GitHub
bryanck opened a new pull request, #12627: URL: https://github.com/apache/iceberg/pull/12627 The rewrite manifests snapshot producer is the only one that does not generate an update event, thus listeners are not notified when a rewrite manifest occurs. This PR adds an update event for rewri

Re: [I] java. lang.UnsupportedOperationException: Unknown delete file content: DATA [iceberg]

2025-04-05 Thread via GitHub
wardlican commented on issue #11981: URL: https://github.com/apache/iceberg/issues/11981#issuecomment-2746740582 > Thanks [@wardlican](https://github.com/wardlican) for reporting this. It looks like this is corrupt metadata, that a DELETE manifest contains actual DATA entries. Do you know w

Re: [I] Cannot scan empty table [iceberg-rust]

2025-04-05 Thread via GitHub
danking commented on issue #1145: URL: https://github.com/apache/iceberg-rust/issues/1145#issuecomment-2776691568 AFAICT, this still fails on `fcc88920f52dbae53257757e2d33825bea4b51a9`. I've modified my test code a bit since then, but it still hits the same error in scan/mod.rs. I do

Re: [PR] Spark: Add some tests for variant fixup [iceberg]

2025-04-05 Thread via GitHub
rdblue commented on code in PR #12497: URL: https://github.com/apache/iceberg/pull/12497#discussion_r2001948430 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestSparkFixupTypes.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Spark: prefix SparkTable with 'iceberg' to clearly identify Iceberg table [iceberg]

2025-04-05 Thread via GitHub
wypoon commented on PR #12543: URL: https://github.com/apache/iceberg/pull/12543#issuecomment-2751698066 > @wypoon , the motivation for this PR is when Iโ€™m trying to capture data lineage using DataHub in spark streaming mode. In the DataHub [code](https://github.com/datahub-project/datahub/

Re: [I] "manylinux_2_34_aarch64" wheel request [iceberg-python]

2025-04-05 Thread via GitHub
kevinjqliu commented on issue #1807: URL: https://github.com/apache/iceberg-python/issues/1807#issuecomment-2755196283 @gabor-one would you like to contribute adding this? heres where to build the wheels https://github.com/apache/iceberg-python/blob/278f7643cd62f9e14496177632cb48d9b52e553d

Re: [PR] Core: Enable row lineage for all v3 tables [iceberg]

2025-04-05 Thread via GitHub
rdblue closed pull request #12593: Core: Enable row lineage for all v3 tables URL: https://github.com/apache/iceberg/pull/12593 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

2025-04-05 Thread via GitHub
Fokko closed issue #1798: Error reading table after appending pyarrow table URL: https://github.com/apache/iceberg-python/issues/1798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] refine: refine ManifestFile [iceberg-rust]

2025-04-05 Thread via GitHub
ZENOTME commented on code in PR #1117: URL: https://github.com/apache/iceberg-rust/pull/1117#discussion_r2006812193 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -590,14 +590,19 @@ impl ManifestFile { self.added_files_count.is_none() || self.added_files_count.unwra

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.30.31 to 2.31.1 [iceberg]

2025-04-05 Thread via GitHub
dependabot[bot] commented on PR #12536: URL: https://github.com/apache/iceberg/pull/12536#issuecomment-2746016004 Superseded by #12621. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] Refactor AwsProperties into separated properties classes [iceberg]

2025-04-05 Thread via GitHub
lliangyu-lin commented on issue #7515: URL: https://github.com/apache/iceberg/issues/7515#issuecomment-2762660767 I can take this up to refactor the glue and lakeformation properties. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Docs: Update block spacing guideline in contribute.md [iceberg]

2025-04-05 Thread via GitHub
nastra commented on code in PR #12641: URL: https://github.com/apache/iceberg/pull/12641#discussion_r2012585502 ## site/docs/contribute.md: ## @@ -422,6 +422,50 @@ Use `this` when assigning values to instance variables, making it clear when the 2. Use `.` to create a hierarchy

Re: [I] Arrow to iceberg schema conversion does not preserve names [iceberg-rust]

2025-04-05 Thread via GitHub
liurenjie1024 commented on issue #1039: URL: https://github.com/apache/iceberg-rust/issues/1039#issuecomment-2772516604 I think this is not a bug as list element name is required by iceberg spec, see https://iceberg.apache.org/spec/#schemas I'll close this for now, feel free to reopen

Re: [PR] Alternative implementation for building and testing hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-05 Thread via GitHub
wypoon commented on code in PR #12721: URL: https://github.com/apache/iceberg/pull/12721#discussion_r2028165803 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java: ## @@ -186,9 +189,9 @@ public void stop() throws Exception { if (executorService !

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-05 Thread via GitHub
deniskuzZ commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2016739545 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -45,22 +49,53 @@ private PartitionStatsUtil() {} * @param table the table for which part

Re: [PR] feat(puffin): Add PuffinWriter [iceberg-rust]

2025-04-05 Thread via GitHub
fqaiser94 commented on code in PR #959: URL: https://github.com/apache/iceberg-rust/pull/959#discussion_r2009143697 ## crates/iceberg/src/writer/file_writer/track_writer.rs: ## @@ -26,16 +26,21 @@ use crate::Result; /// `TrackWriter` is used to track the written size. pub(crat

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-05 Thread via GitHub
pvary commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2012517858 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java: ## @@ -0,0 +1,264 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Core, Hive: Double check commit status in case of commit conflict for NoLock [iceberg]

2025-04-05 Thread via GitHub
pvary commented on code in PR #12637: URL: https://github.com/apache/iceberg/pull/12637#discussion_r2012254481 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreExtension.java: ## @@ -40,16 +40,10 @@ private HiveMetastoreExtension(String databaseName, Map hiv

Re: [PR] Flink: Support create table like and source watermark for flink sql to 1.18,1.19 [iceberg]

2025-04-05 Thread via GitHub
pvary commented on PR #12643: URL: https://github.com/apache/iceberg/pull/12643#issuecomment-2761943719 I still prefer 1 by 1 backports. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] CORE: Allow HTTPClient to parse headers from properties. [iceberg]

2025-04-05 Thread via GitHub
wolflex888 commented on code in PR #12595: URL: https://github.com/apache/iceberg/pull/12595#discussion_r2017814537 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -332,6 +336,41 @@ public void testInitializeWithBadArguments() throws IOException {

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-05 Thread via GitHub
yingcai-cy commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2024469563 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] CORE: Allow HTTPClient to parse headers from properties. [iceberg]

2025-04-05 Thread via GitHub
ajantha-bhat commented on PR #12595: URL: https://github.com/apache/iceberg/pull/12595#issuecomment-2774086374 I think this is the last PR needed for 1.9.0 release. Happy to help in getting it merged. I think we are pretty close. -- This is an automated message from the Apache Git Servic

[PR] Build: Retry flaky test [iceberg]

2025-04-05 Thread via GitHub
manuzhang opened a new pull request, #12707: URL: https://github.com/apache/iceberg/pull/12707 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-04-05 Thread via GitHub
xiaoxuandev commented on code in PR #12671: URL: https://github.com/apache/iceberg/pull/12671#discussion_r2022170991 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -290,6 +290,10 @@ public class S3FileIOProperties implements Serializable { pub

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-04-05 Thread via GitHub
rdblue commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r2008315387 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +565,148 @@ public int hashCode() { } } + public static class GeometryType extends Pr

Re: [PR] WIP Parquet: Support reading/writing geometry and geography columns [iceberg]

2025-04-05 Thread via GitHub
github-actions[bot] closed pull request #12347: WIP Parquet: Support reading/writing geometry and geography columns URL: https://github.com/apache/iceberg/pull/12347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

  1   2   3   >