Re: [I] Issue in Reading Iceberg tables in Nessie + Minio using Pyiceberg [iceberg-python]

heman026 commented on issue #1560: URL: https://github.com/apache/iceberg-python/issues/1560#issuecomment-2712187489 This is my working setup. Configuration of Nessie might differ for you. https://github.com/apache/iceberg-python/issues/1524#issuecomment-2683847774 Can you check if thi

Re: [I] org.apache.thrift.transport.TSaslTransport: SASL negotiation failure [iceberg]

github-actions[bot] closed issue #11019: org.apache.thrift.transport.TSaslTransport: SASL negotiation failure URL: https://github.com/apache/iceberg/issues/11019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Issue during Upsert [iceberg-python]

mattmartin14 commented on issue #1759: URL: https://github.com/apache/iceberg-python/issues/1759#issuecomment-2712180183 I encountered this issue when writing the upsert method. If your join column is a single column, then the function scales fine and won't hit a recursion limit. I stresse

Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

tomtongue commented on issue #7160: URL: https://github.com/apache/iceberg/issues/7160#issuecomment-2712820271 @nastra I believe the remaining migration is only in Spark 3.4. If possible, can I proceed them to the JUnit 5 with AssertJ style? (and, I think this also should help the backport

Re: [PR] Build: Bump getdaft from 0.4.4 to 0.4.6 [iceberg-python]

dependabot[bot] closed pull request #1758: Build: Bump getdaft from 0.4.4 to 0.4.6 URL: https://github.com/apache/iceberg-python/pull/1758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] Core: FileRewritePlanner implementation [iceberg]

pvary opened a new pull request, #12493: URL: https://github.com/apache/iceberg/pull/12493 This PR creates the core implementations for the Planners defined by #12306. The PR deprecates the old `*Rewriter` implementations. Adds test for the Planners. This is part of the #11513

Re: [I] Flink: add more sink shuffling support [iceberg]

github-actions[bot] commented on issue #6303: URL: https://github.com/apache/iceberg/issues/6303#issuecomment-2712134983 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Ignore `paste` in audit check. [iceberg-rust]

liurenjie1024 closed issue #1062: Ignore `paste` in audit check. URL: https://github.com/apache/iceberg-rust/issues/1062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] chore: Ignore paste crate in rust crate audit check. [iceberg-rust]

liurenjie1024 merged PR #1063: URL: https://github.com/apache/iceberg-rust/pull/1063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988124358 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988128545 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [I] Ignore `paste` in audit check. [iceberg-rust]

liurenjie1024 closed issue #1062: Ignore `paste` in audit check. URL: https://github.com/apache/iceberg-rust/issues/1062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] `expire_snapshots` sudden increase in compute/memory usage [iceberg]

github-actions[bot] closed issue #11017: `expire_snapshots` sudden increase in compute/memory usage URL: https://github.com/apache/iceberg/issues/11017 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Core, Spark 3.5: Apply Equality Deletes when Doing Copy on Write [iceberg]

wypoon commented on PR #12479: URL: https://github.com/apache/iceberg/pull/12479#issuecomment-2712318261 I checked out this change and ran the newly added test with and without the fix. The behavior is as expected, except that even without the fix, two of the six cases in the parameterized

Re: [I] Support full table scanning of partitioned table is prohibited [iceberg]

ebyhr commented on issue #12474: URL: https://github.com/apache/iceberg/issues/12474#issuecomment-2712627254 Trino already supports `iceberg.query-partition-filter-required` config property and `query_partition_filter_required` session property. -- This is an automated message from the A

Re: [I] org.apache.thrift.transport.TSaslTransport: SASL negotiation failure [iceberg]

github-actions[bot] commented on issue #11019: URL: https://github.com/apache/iceberg/issues/11019#issuecomment-2712135284 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Spec and Doc inconsistency about supported schema evolution for list and map types [iceberg]

github-actions[bot] closed issue #11020: Spec and Doc inconsistency about supported schema evolution for list and map types URL: https://github.com/apache/iceberg/issues/11020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Rewrite manifests [iceberg-python]

Fokko commented on code in PR #1661: URL: https://github.com/apache/iceberg-python/pull/1661#discussion_r1987938857 ## pyiceberg/table/update/snapshot.py: ## @@ -477,6 +478,20 @@ def _deleted_entries(self) -> List[ManifestEntry]: return [] +@dataclass(init=False) R

Re: [I] Accessing Minio with Pyiceberg [iceberg]

github-actions[bot] closed issue #10709: Accessing Minio with Pyiceberg URL: https://github.com/apache/iceberg/issues/10709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Location Ownership [iceberg]

github-actions[bot] commented on issue #9133: URL: https://github.com/apache/iceberg/issues/9133#issuecomment-2712135088 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Parquet: Fix Reader leak by removing useless copy [iceberg]

github-actions[bot] commented on PR #12079: URL: https://github.com/apache/iceberg/pull/12079#issuecomment-2712135448 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Support full table scanning of partitioned table is prohibited [iceberg]

melin commented on issue #12474: URL: https://github.com/apache/iceberg/issues/12474#issuecomment-2712196359 spark extensions: https://github.com/apache/kyuubi/blob/master/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/watchdog/MaxScanStrategy.scala -- Th

Re: [PR] feat: (catalog/glue) Add support for CreateTable [iceberg-go]

zeroshade commented on code in PR #326: URL: https://github.com/apache/iceberg-go/pull/326#discussion_r1987873141 ## catalog/glue/glue_test.go: ## @@ -778,12 +781,103 @@ func TestGlueListNamespacesIntegration(t *testing.T) { } assert := require.New(t) - a

Re: [PR] Flink: Support source watermark for flink sql windows [iceberg]

pvary commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1988454071 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/source/TestSqlBase.java: ## @@ -75,6 +77,18 @@ protected TableEnvironment getTableEnv() { return tEnv;

Re: [I] ICEBERG performance is slow when querying tables with a large number of partitions. [iceberg]

github-actions[bot] commented on issue #8161: URL: https://github.com/apache/iceberg/issues/8161#issuecomment-2712135024 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] ICEBERG performance is slow when querying tables with a large number of partitions. [iceberg]

github-actions[bot] closed issue #8161: ICEBERG performance is slow when querying tables with a large number of partitions. URL: https://github.com/apache/iceberg/issues/8161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] Type Promotion: Int/Long to String [iceberg]

github-actions[bot] commented on issue #9064: URL: https://github.com/apache/iceberg/issues/9064#issuecomment-2712135037 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] `expire_snapshots` sudden increase in compute/memory usage [iceberg]

github-actions[bot] commented on issue #11017: URL: https://github.com/apache/iceberg/issues/11017#issuecomment-2712135261 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Missing records after compaction using `rewrite_data_files` [iceberg]

github-actions[bot] closed issue #11014: Missing records after compaction using `rewrite_data_files` URL: https://github.com/apache/iceberg/issues/11014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Accessing Minio with Pyiceberg [iceberg]

github-actions[bot] commented on issue #10709: URL: https://github.com/apache/iceberg/issues/10709#issuecomment-2712135142 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Parquet: Fix Reader leak by removing useless copy [iceberg]

github-actions[bot] closed pull request #12079: Parquet: Fix Reader leak by removing useless copy URL: https://github.com/apache/iceberg/pull/12079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Spark 3.5, Core: Supplement test case for metadata_log_entries after expire snapshot [iceberg]

github-actions[bot] commented on PR #11901: URL: https://github.com/apache/iceberg/pull/11901#issuecomment-2712135392 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Spec and Doc inconsistency about supported schema evolution for list and map types [iceberg]

github-actions[bot] commented on issue #11020: URL: https://github.com/apache/iceberg/issues/11020#issuecomment-2712135309 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes [iceberg]

RussellSpitzer commented on issue #12467: URL: https://github.com/apache/iceberg/issues/12467#issuecomment-2711263123 Getting closer, the problem is that "ignoreResiduals" is actually being ignored (set to false) when we actually perform the read. Trying to figure out why. -- This is an

Re: [PR] Build: Bump testcontainers from 1.20.5 to 1.20.6 [iceberg]

nastra closed pull request #12484: Build: Bump testcontainers from 1.20.5 to 1.20.6 URL: https://github.com/apache/iceberg/pull/12484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Build: Bump com.gorylenko.gradle-git-properties:gradle-git-properties from 2.4.2 to 2.5.0 [iceberg]

nastra merged PR #12486: URL: https://github.com/apache/iceberg/pull/12486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump testcontainers from 1.20.5 to 1.20.6 [iceberg]

dependabot[bot] commented on PR #12484: URL: https://github.com/apache/iceberg/pull/12484#issuecomment-2709714172 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988117138 ## core/src/main/java/org/apache/iceberg/FieldMetrics.java: ## @@ -27,6 +27,18 @@ public class FieldMetrics { private final T lowerBound; private final T upperBo

Re: [PR] [Do not merge] Iterative `bind` with a stack instead of recursion [iceberg-python]

smaheshwar-pltr commented on code in PR #1783: URL: https://github.com/apache/iceberg-python/pull/1783#discussion_r1988154657 ## tests/table/test_upsert.py: ## @@ -408,6 +408,31 @@ def test_upsert_into_empty_table(catalog: Catalog) -> None: assert upd.rows_inserted == 4

Re: [PR] [Do not merge] Iterative `bind` with a stack instead of recursion [iceberg-python]

smaheshwar-pltr commented on code in PR #1783: URL: https://github.com/apache/iceberg-python/pull/1783#discussion_r1988154986 ## tests/table/test_upsert.py: ## @@ -408,6 +408,31 @@ def test_upsert_into_empty_table(catalog: Catalog) -> None: assert upd.rows_inserted == 4

Re: [I] Spark can't get information from metadata tables [iceberg]

varpa89 commented on issue #12466: URL: https://github.com/apache/iceberg/issues/12466#issuecomment-2709868284 Thank you @singhpk234 Yes rest catalog that we use returns BadRequestException because we don't allow namespaces with dots If we disable validation in our metastore, NoSu

Re: [PR] Build: Bump testcontainers from 1.20.5 to 1.20.6 [iceberg]

nastra merged PR #12484: URL: https://github.com/apache/iceberg/pull/12484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

smaheshwar-pltr commented on PR #433: URL: https://github.com/apache/iceberg-python/pull/433#issuecomment-2710268654 > Are you still working on this PR? For context, we are doing some interesting (and probably wrong) stuff to mimic Iceberg Java's REPLACE TABLE in https://github.com/bodo-ai/

Re: [I] Issue during Upsert [iceberg-python]

Fokko commented on issue #1759: URL: https://github.com/apache/iceberg-python/issues/1759#issuecomment-2711738238 Thanks everyone, for raising this. The number of columns that you upsert on can cause problems since each column will duplicate the complexity of the predicate. As a in

Re: [PR] chore(deps): Bump either from 1.13.0 to 1.15.0 [iceberg-rust]

liurenjie1024 merged PR #1060: URL: https://github.com/apache/iceberg-rust/pull/1060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988118605 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java: ## @@ -142,7 +144,8 @@ public void add(T value) { public Metrics metrics() { Preconditi

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988121235 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988121875 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988122364 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [I] Transactions do not support Upsert [iceberg-python]

Fokko commented on issue #1776: URL: https://github.com/apache/iceberg-python/issues/1776#issuecomment-2711732251 Thanks @suarez-agu for raising this. The transaction should also support upsert. Are you interested in contributing this? -- This is an automated message from the Apache Git

Re: [PR] Parquet: Implement Variant metrics [iceberg]

rdblue commented on code in PR #12496: URL: https://github.com/apache/iceberg/pull/12496#discussion_r1988123170 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java: ## @@ -0,0 +1,613 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] doc: run doc test [iceberg-rust]

Fokko commented on code in PR #1066: URL: https://github.com/apache/iceberg-rust/pull/1066#discussion_r1986962950 ## crates/iceberg/src/writer/mod.rs: ## @@ -26,23 +26,58 @@ //! 2. IcebergWriter: Focus on the logical format of iceberg table. It will write the data using the Fi

Re: [PR] Build: Rename versions.spark.hive3* to versions.spark3* [iceberg]

Fokko commented on PR #12489: URL: https://github.com/apache/iceberg/pull/12489#issuecomment-2710028853 Thanks for fixing this @pan3793, and thanks for the review @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Rename versions.spark.hive3* to versions.spark3* [iceberg]

Fokko merged PR #12489: URL: https://github.com/apache/iceberg/pull/12489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Auth Manager API part 6: API enablement [iceberg]

adutra commented on PR #12197: URL: https://github.com/apache/iceberg/pull/12197#issuecomment-2710091175 @nastra @danielcweeks can I get another review please? 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Delete a row (primary key ID) from iceberg table [iceberg]

Fokko commented on issue #12490: URL: https://github.com/apache/iceberg/issues/12490#issuecomment-2711847568 Hey @Shekharrajak Since snapshots are immutable, the only way to do this, is to create a new snapshot without the row. Then, [expire all the old snapshots](https://iceberg.apache.org

Re: [PR] Spark: 4.0 snapshot support [iceberg]

huaxingao commented on PR #11583: URL: https://github.com/apache/iceberg/pull/11583#issuecomment-2711818415 It's much easier to create a new PR than rebasing this one. I will close this PR for now and create a new one instead. -- This is an automated message from the Apache Git Service.

Re: [PR] Fix partitioning on timestamp field with year and month transformations [iceberg]

Fokko commented on PR #12443: URL: https://github.com/apache/iceberg/pull/12443#issuecomment-2711858719 I'm going ahead and closing this one since the error is intentional. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Fix partitioning on timestamp field with year and month transformations [iceberg]

Fokko closed pull request #12443: Fix partitioning on timestamp field with year and month transformations URL: https://github.com/apache/iceberg/pull/12443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark: 4.0 snapshot support [iceberg]

huaxingao closed pull request #11583: Spark: 4.0 snapshot support URL: https://github.com/apache/iceberg/pull/11583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] How to understand "Partition evolution is a metadata operation and does not eagerly rewrite files." [iceberg]

RussellSpitzer commented on issue #12492: URL: https://github.com/apache/iceberg/issues/12492#issuecomment-2710412030 It will not. New files will be written to the new spec, old files will remain in the old spec. Files from multiple partition specs can always exist in the same table.

[PR] Build: Bump mkdocstrings from 0.28.2 to 0.29.0 [iceberg-python]

dependabot[bot] opened a new pull request, #1781: URL: https://github.com/apache/iceberg-python/pull/1781 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.28.2 to 0.29.0. Release notes Sourced from https://github.com/mkdocstrings/mkdocstrings/releases";>mkd

Re: [PR] Build: Bump getdaft from 0.4.4 to 0.4.6 [iceberg-python]

dependabot[bot] commented on PR #1758: URL: https://github.com/apache/iceberg-python/pull/1758#issuecomment-2711972030 Superseded by #1780. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] Build: Bump mkdocs-autorefs from 1.4.0 to 1.4.1 [iceberg-python]

dependabot[bot] opened a new pull request, #1782: URL: https://github.com/apache/iceberg-python/pull/1782 Bumps [mkdocs-autorefs](https://github.com/mkdocstrings/autorefs) from 1.4.0 to 1.4.1. Release notes Sourced from https://github.com/mkdocstrings/autorefs/releases";>mkdocs-aut

[I] Unable to set `write-default` for a column [iceberg]

malhotrashivam opened a new issue, #12495: URL: https://github.com/apache/iceberg/issues/12495 ### Apache Iceberg version 1.8.1 (latest release) ### Query engine None ### Please describe the bug 🐞 - I am attempting to create a table via the Iceberg Java API

Re: [PR] chore(deps): Bump tempfile from 3.17.1 to 3.18.0 [iceberg-rust]

liurenjie1024 merged PR #1059: URL: https://github.com/apache/iceberg-rust/pull/1059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Issue during Upsert [iceberg-python]

smaheshwar-pltr commented on issue #1759: URL: https://github.com/apache/iceberg-python/issues/1759#issuecomment-2712089897 Have a test that I think repros this in (draft) https://github.com/apache/iceberg-python/pull/1783. Also, (draft) https://github.com/apache/iceberg-python/pull/