[I] Kafka Connect config providers are not working when CVE-2024-31141 fix is applied [iceberg]

2025-02-10 Thread via GitHub
AnatolyPopov opened a new issue, #12221: URL: https://github.com/apache/iceberg/issues/12221 ### Apache Iceberg version None ### Query engine Kafka Connect ### Please describe the bug 🐞 When a Kafka Connect worker is configured with secret providers and the

Re: [PR] Update documentation / add missing Iceberg table read properties [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on code in PR #12163: URL: https://github.com/apache/iceberg/pull/12163#discussion_r1950361539 ## docs/docs/configuration.md: ## @@ -26,114 +26,117 @@ Iceberg tables support table properties to configure table behavior, like the de ### Read properties -

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1950358255 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics of

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1950358255 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics of

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1950356735 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics of

[I] Improve ThreadPools for graceful shutdown [iceberg]

2025-02-10 Thread via GitHub
ochanism opened a new issue, #12220: URL: https://github.com/apache/iceberg/issues/12220 ### Feature Request / Improvement I implemented JAVA server programs to manage Iceberg tables using Iceberg JAVA SDK (1.7.1). But I've been suffering from an ungraceful shutdown issue. The s

Re: [PR] Data: Handle case where partition location is missing for `TableMigrationUtil` [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on PR #12212: URL: https://github.com/apache/iceberg/pull/12212#issuecomment-264924 @jshmchenxi Can we add an end-to-end test in `TestSnapshotTableAction`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-10 Thread via GitHub
geruh opened a new pull request, #1642: URL: https://github.com/apache/iceberg-python/pull/1642 Adding support for writing metadata to a custom path set via `write.metadata.path` property. Since the Python library consolidates the table operation classes in both the table and catalog classe

Re: [PR] Data: Handle case where partition location is missing for `TableMigrationUtil` [iceberg]

2025-02-10 Thread via GitHub
jshmchenxi commented on PR #12212: URL: https://github.com/apache/iceberg/pull/12212#issuecomment-2649962891 @manuzhang @RussellSpitzer Thanks for the suggestion! I've added test cases to cover this change. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Docs: Add missing types to the spec v3 summary [iceberg]

2025-02-10 Thread via GitHub
wgtmac commented on PR #12219: URL: https://github.com/apache/iceberg/pull/12219#issuecomment-2649955064 cc @szehon-ho @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] Docs: Add missing types to the spec v3 summary [iceberg]

2025-02-10 Thread via GitHub
wgtmac opened a new pull request, #12219: URL: https://github.com/apache/iceberg/pull/12219 variant and geospatial types are recently added to the v3 spec but the summary is not updated in sync. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-10 Thread via GitHub
ZENOTME commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2649905192 > @Fokko I was looking to do the name mapping and more metrics in another pr. Would you rather I include it in this one? ``` def pyarrow_to_schema( schema: pa.Schema, n

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-10 Thread via GitHub
ZENOTME commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2649902294 > while the ArrowFileReader provides the parsed metadata which doesn't contain enough information Thanks for your investigation @jonathanc-n! But seems the parsed metadata cont

Re: [PR] Concurrent table scans [iceberg-rust]

2025-02-10 Thread via GitHub
xxchan commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1950284548 ## crates/iceberg/src/scan.rs: ## @@ -199,134 +219,119 @@ impl<'a> TableScanBuilder<'a> { field_ids.push(field_id); } -Ok(TableScan {

Re: [PR] Core: Fix move/update/makeRequire/makeOptional fail after rename schema (#10830) [iceberg]

2025-02-10 Thread via GitHub
imtzer commented on code in PR #12202: URL: https://github.com/apache/iceberg/pull/12202#discussion_r1950228939 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -391,13 +391,14 @@ private Integer findForMove(String name) { if (addedId != null) { retu

[PR] fix(REST): Misleading error messages in `iceberg-catalog-rest` and allow `StatusCode::OK` in responses [iceberg-rust]

2025-02-10 Thread via GitHub
connortsui20 opened a new pull request, #962: URL: https://github.com/apache/iceberg-rust/pull/962 Several of the error messages in `iceberg-catalog-rest` had misleading error messages, so this PR fixes some of those. Additionally, several of the `Catalog` methods on `RestCatalog` do

Re: [PR] ci(dependabot): fix dependabot config [iceberg-go]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #299: URL: https://github.com/apache/iceberg-go/pull/299#discussion_r1950190389 ## .github/dependabot.yml: ## @@ -25,3 +27,4 @@ updates: groups: gomod_updates: applies-to: version-updates +dependency-type: productio

Re: [PR] Docs: Fix links to Javadoc [iceberg]

2025-02-10 Thread via GitHub
manuzhang commented on PR #12215: URL: https://github.com/apache/iceberg/pull/12215#issuecomment-2649633151 @amogh-jahagirdar please pay attention to this link when you release 1.8.0 docs. I'm not sure if it has been fixed in the release script. -- This is an automated message from the Ap

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-10 Thread via GitHub
danielcweeks commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1950100966 ## api/src/main/java/org/apache/iceberg/UpdateSchema.java: ## @@ -280,6 +410,30 @@ default UpdateSchema updateColumn(String name, Type.PrimitiveType newType, St

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-10 Thread via GitHub
danielcweeks commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1950100200 ## api/src/main/java/org/apache/iceberg/UpdateSchema.java: ## @@ -280,6 +410,30 @@ default UpdateSchema updateColumn(String name, Type.PrimitiveType newType, St

Re: [PR] Core: Bulk deletion in RemoveSnapshots [iceberg]

2025-02-10 Thread via GitHub
amogh-jahagirdar commented on code in PR #11837: URL: https://github.com/apache/iceberg/pull/11837#discussion_r1950075730 ## core/src/main/java/org/apache/iceberg/BulkDeleteConsumer.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-10 Thread via GitHub
jonathanc-n commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2649571942 For metadata retrieval, it seems i can use `ParquetWriter::to_data_file_builder` to avoid duplicating more code. However the file metadata it takes in is the thrift `FileMetadata`

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-10 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1950087975 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogTable.java: ## @@ -339,11 +339,6 @@ public void testAlterTableAddColumn() {

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-10 Thread via GitHub
hantangwangd commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1950079317 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java: ## @@ -57,7 +60,10 @@ protected static Object[][] parameters() { {

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-10 Thread via GitHub
hantangwangd commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1950073181 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -233,6 +233,8 @@ public static List mergeTasks(List tasks) { } public static lo

Re: [PR] Core: Bulk deletion in RemoveSnapshots [iceberg]

2025-02-10 Thread via GitHub
amogh-jahagirdar commented on PR #11837: URL: https://github.com/apache/iceberg/pull/11837#issuecomment-2649544429 Sorry for the late followup, I'm taking a look! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Support custom spark procedure in plugin mode for iceberg [iceberg]

2025-02-10 Thread via GitHub
github-actions[bot] commented on issue #10906: URL: https://github.com/apache/iceberg/issues/10906#issuecomment-2649541197 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Refactor TestIcebergCommitter state recovery unit tests to use checkpointId=1 [iceberg]

2025-02-10 Thread via GitHub
github-actions[bot] commented on issue #10942: URL: https://github.com/apache/iceberg/issues/10942#issuecomment-2649541253 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Iceberg materialized view [iceberg]

2025-02-10 Thread via GitHub
github-actions[bot] commented on issue #10890: URL: https://github.com/apache/iceberg/issues/10890#issuecomment-2649541150 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Support committed callback [iceberg]

2025-02-10 Thread via GitHub
github-actions[bot] commented on issue #10936: URL: https://github.com/apache/iceberg/issues/10936#issuecomment-2649541231 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Spark: Support singular form of years, months, days, and hours functions [iceberg]

2025-02-10 Thread via GitHub
wypoon commented on PR #12117: URL: https://github.com/apache/iceberg/pull/12117#issuecomment-2649532502 @RussellSpitzer it's for consistency. As you know, the spec specifies `year`, `month`, `day` and `hour` for the partition transforms. Originally, Spark only supported the plural forms `y

Re: [PR] Core: Fix move/update/makeRequire/makeOptional fail after rename schema (#10830) [iceberg]

2025-02-10 Thread via GitHub
ebyhr commented on code in PR #12202: URL: https://github.com/apache/iceberg/pull/12202#discussion_r1950062665 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -391,13 +391,14 @@ private Integer findForMove(String name) { if (addedId != null) { retur

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950061705 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[PR] Build: skip scheduled docker image publish workflows on forks [iceberg]

2025-02-10 Thread via GitHub
dramaticlly opened a new pull request, #12218: URL: https://github.com/apache/iceberg/pull/12218 Only enable workflow #11632 in apache and avoid failure like https://github.com/dramaticlly/iceberg/actions/runs/13127287689/job/36625899396 @nastra @Fokko @sungwy can you help review? -

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649504621 @mattmartin14 i tried to fix as many linter issue as possible, the changes are in https://github.com/apache/iceberg-python/compare/main...kevinjqliu:iceberg-python:kevinjqliu/15

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950048200 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950039374 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950023950 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950036853 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[PR] Docs: Add documentation for Rate limiting in Spark Structured Streaming [iceberg]

2025-02-10 Thread via GitHub
singhpk234 opened a new pull request, #12217: URL: https://github.com/apache/iceberg/pull/12217 revives PR: https://github.com/apache/iceberg/pull/8044 Request for adding this in docs : Past occurrence : [1] https://apache-iceberg.slack.com/archives/C025PH0G1D4/p171674020992942

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950032898 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Build: Bump coverage from 7.6.10 to 7.6.11 [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu merged PR #1641: URL: https://github.com/apache/iceberg-python/pull/1641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-10 Thread via GitHub
RussellSpitzer commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1950014100 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java: ## @@ -57,7 +60,10 @@ protected static Object[][] parameters() { {

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950014481 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,273 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[PR] Build: Bump coverage from 7.6.10 to 7.6.11 [iceberg-python]

2025-02-10 Thread via GitHub
dependabot[bot] opened a new pull request, #1641: URL: https://github.com/apache/iceberg-python/pull/1641 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.6.10 to 7.6.11. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's c

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-10 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1950004069 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,273 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Implement update for `remove-snapshot-ref` action [iceberg-python]

2025-02-10 Thread via GitHub
grihabor commented on PR #1598: URL: https://github.com/apache/iceberg-python/pull/1598#issuecomment-2649415003 Thank you! It passed now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark: Support singular form of years, months, days, and hours functions [iceberg]

2025-02-10 Thread via GitHub
RussellSpitzer commented on PR #12117: URL: https://github.com/apache/iceberg/pull/12117#issuecomment-2649381298 What's the benefit of being able to use both singular and plural forms? Seems like we are just opening up the api to allow 2 paths for the same thing? -- This is an automated m

Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

2025-02-10 Thread via GitHub
dramaticlly commented on issue #12134: URL: https://github.com/apache/iceberg/issues/12134#issuecomment-2649328210 > I added this on the mailing list but thought I'd drop it here as well: > > I think the issue here is that you're using the commit api in table operations to perform a n

Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

2025-02-10 Thread via GitHub
danielcweeks commented on issue #12134: URL: https://github.com/apache/iceberg/issues/12134#issuecomment-2649320278 I added this on the mailing list but thought I'd drop it here as well: I think the issue here is that you're using the commit api in table operations to perform a non-in

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649313405 i was able to run: ```bash ruff check --fix ``` hope that helped -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649305878 @kevinjqliu - also, i'm seeing that the CI is failing to pull datafusion; how do i fix that, since i"m using datafusion in my test files? I did run the command: ```bas

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649303687 > Looks like CI failed due to the linter, can you run `make lint` locally? > > > i updated my original description to have a link to PR 402; > > It would be great

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#discussion_r1949930665 ## mkdocs/docs/api.md: ## @@ -1533,3 +1533,141 @@ df.show(2) (Showing first 2 rows) ``` + +### Polars + +PyIceberg interfaces closely with Polars Datafram

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2649301216 can you rebase off main? looks like theres a conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-10 Thread via GitHub
Fokko commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2649298184 @jonathanc-n Let's do that in a separate PR 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-10 Thread via GitHub
dttung2905 commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1949932455 ## catalog/rest/rest.go: ## @@ -989,3 +989,54 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, ni

Re: [I] software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. [iceberg]

2025-02-10 Thread via GitHub
petartushev commented on issue #11997: URL: https://github.com/apache/iceberg/issues/11997#issuecomment-2649285762 @timoha upgraded the hadoop dependencies and my `pom.xml` file now looks like this: ``` org.apache.spark spark-c

Re: [PR] Materialized View Spec [iceberg]

2025-02-10 Thread via GitHub
stevenzwu commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1949898537 ## format/view-spec.md: ## @@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields: | _required_ | `representations` | A list of [

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649278852 Looks like CI failed due to the linter, can you run `make lint` locally? > i updated my original description to have a link to PR 402; It would be great to rewrite

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649271636 i saw there was an error on the ci on trying to pull in datafusion; i'm using datafusion for my test cases, which i was told is fine. It is used to build the test datasets. Is

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649252673 > Thanks! a couple conflicts from rebasing. would be good to address those before running CI. > > Could you also edit the PR description? And include references to #402

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649246609 Thanks! a couple conflicts from rebasing. would be good to address those before running CI. Could you also edit the PR description? And include references to #402 -- T

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1949895088 ## pyiceberg/expressions/literals.py: ## @@ -29,6 +29,7 @@ from math import isnan from typing import Any, Generic, Type from uuid import UUID +from datet

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-10 Thread via GitHub
yigal-rozenberg commented on code in PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#discussion_r1949896700 ## mkdocs/docs/api.md: ## @@ -1533,3 +1533,111 @@ df.show(2) (Showing first 2 rows) ``` + +### Polars + +PyIceberg interfaces closely with Polars Dat

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649245843 please let me know if you all have any additional issues on the workflows. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-10 Thread via GitHub
yigal-rozenberg commented on code in PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#discussion_r1949880005 ## pyiceberg/table/__init__.py: ## @@ -1624,6 +1638,19 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arr

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649242299 @Fokko , @bitsondatadev and i resolved the conflicts (i think 😀). You should be good to go on kicking off the workflows; we are rebased to the current main branch -- This i

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1949892377 ## pyiceberg/expressions/literals.py: ## @@ -29,6 +29,7 @@ from math import isnan from typing import Any, Generic, Type from uuid import UUID +from datetim

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1949893406 ## pyproject.toml: ## @@ -1409,4 +1409,4 @@ module = "tenacity.*" ignore_missing_imports = true [tool.coverage.run] -source = ['pyiceberg/'] +source = ['p

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-10 Thread via GitHub
RussellSpitzer commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1949883612 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -233,6 +233,8 @@ public static List mergeTasks(List tasks) { } public static

Re: [I] Extract more metadata using `ArrowFileReader` [iceberg-rust]

2025-02-10 Thread via GitHub
jonathanc-n closed issue #961: Extract more metadata using `ArrowFileReader` URL: https://github.com/apache/iceberg-rust/issues/961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Core: Bulk deletion in RemoveSnapshots [iceberg]

2025-02-10 Thread via GitHub
pvary commented on code in PR #11837: URL: https://github.com/apache/iceberg/pull/11837#discussion_r1949864076 ## core/src/main/java/org/apache/iceberg/BulkDeleteConsumer.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[I] Extract more metadata using `ArrowFileReader` [iceberg-rust]

2025-02-10 Thread via GitHub
jonathanc-n opened a new issue, #961: URL: https://github.com/apache/iceberg-rust/issues/961 General idea is to allow ArrowFileReader to store more parquet metadata in its FileMetadata. Part of #960, just to allow for more metadata to construct the `DataFile` Can be found here: [

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
Fokko commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649172716 @mattmartin14 There are conflicts with the main branch, which prohibits me from merging this: https://github.com/user-attachments/assets/b842a6f4-807b-429e-be56-17346d5ac53b";

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649131822 > @mattmartin14 We need to get the tests green before we can merge this into main @Fokko - my tests are all green; is there something else i need to do at the moment? o

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
Fokko commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649130857 Let me add my preferred way of doing this: ``` git remote add github g...@github.com:apache/iceberg-python.git git fetch --all git pull github main git push origin m

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
Fokko commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649129009 @mattmartin14 We need to get the tests green before we can merge this into main -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649124452 > @mattmartin14 you need to rebase off of the `upstream` repo and push to `origin`. Assuming that `upstream` is the `apache/iceberg` repository and `origin` is `mattmartin14/i

Re: [PR] Spec: Support geo type [iceberg]

2025-02-10 Thread via GitHub
szehon-ho commented on PR #10981: URL: https://github.com/apache/iceberg/pull/10981#issuecomment-2649098406 Merged, thanks so much to all who have contributed to this massive discussion and effort ! -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Spec: Support geo type [iceberg]

2025-02-10 Thread via GitHub
szehon-ho merged PR #10981: URL: https://github.com/apache/iceberg/pull/10981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
bitsondatadev commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2649081273 @mattmartin14 you need to rebase off of the `upstream` repo and push to `origin`. Assuming that `upstream` is the `apache/iceberg` repository and `origin` is `mattmartin14/ic

Re: [PR] Repo: Set status checks to strict [iceberg-python]

2025-02-10 Thread via GitHub
Fokko merged PR #1640: URL: https://github.com/apache/iceberg-python/pull/1640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-10 Thread via GitHub
jonathanc-n commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2649038372 @Fokko I was looking to do the name mapping and more metrics in another pr. Would you rather I include it in this one? -- This is an automated message from the Apache Git Service

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-10 Thread via GitHub
dttung2905 commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1949750836 ## catalog/rest/rest.go: ## @@ -989,3 +989,54 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, ni

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-10 Thread via GitHub
zeroshade commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1949747930 ## catalog/rest/rest.go: ## @@ -989,3 +989,54 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, nil

Re: [PR] chore: improve and fix the rest example [iceberg-rust]

2025-02-10 Thread via GitHub
connortsui20 commented on code in PR #842: URL: https://github.com/apache/iceberg-rust/pull/842#discussion_r1949747913 ## crates/examples/src/rest_catalog_table.rs: ## @@ -21,11 +21,16 @@ use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; use iceberg::{Catalog, Tab

Re: [PR] feat(cmd/iceberg): Add commands for iceberg CLI. [iceberg-go]

2025-02-10 Thread via GitHub
zeroshade merged PR #298: URL: https://github.com/apache/iceberg-go/pull/298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Improve help text for CLI [iceberg-go]

2025-02-10 Thread via GitHub
zeroshade closed issue #61: Improve help text for CLI URL: https://github.com/apache/iceberg-go/issues/61 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] Improve help text for CLI [iceberg-go]

2025-02-10 Thread via GitHub
zeroshade closed issue #61: Improve help text for CLI URL: https://github.com/apache/iceberg-go/issues/61 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] Break circular import [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu merged PR #1639: URL: https://github.com/apache/iceberg-python/pull/1639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-10 Thread via GitHub
dttung2905 commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1949738471 ## catalog/rest/rest.go: ## @@ -989,3 +989,54 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, ni

Re: [PR] chore: tweak dependabot to bundle all go mod upgrades into the same PR [iceberg-go]

2025-02-10 Thread via GitHub
zeroshade commented on PR #289: URL: https://github.com/apache/iceberg-go/pull/289#issuecomment-2648978253 I very much appreciate the assistance! Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Repo: Set status checks to strict [iceberg-python]

2025-02-10 Thread via GitHub
Fokko commented on code in PR #1640: URL: https://github.com/apache/iceberg-python/pull/1640#discussion_r1949717498 ## .asf.yaml: ## @@ -34,6 +34,10 @@ github: rebase: true protected_branches: main: + required_status_checks: +# strict means "Require bra

Re: [PR] chore: tweak dependabot to bundle all go mod upgrades into the same PR [iceberg-go]

2025-02-10 Thread via GitHub
raidancampbell commented on PR #289: URL: https://github.com/apache/iceberg-go/pull/289#issuecomment-2648948145 Sounds good: Feel free to bother me if it doesn't end up working how you'd like. I saw what appeared to be some tedious work with dependency upgrades and am looking to make it ea

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-10 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2648953956 > Could you rebase off the main to pick up the latest set of changes? > > Alternatively, I'm happy to help push out the fix for poetry lock and pyproject if needed

Re: [PR] Repo: Set status checks to strict [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #1640: URL: https://github.com/apache/iceberg-python/pull/1640#discussion_r1949703260 ## .asf.yaml: ## @@ -34,6 +34,10 @@ github: rebase: true protected_branches: main: + required_status_checks: +# strict means "Requir

Re: [PR] Repo: Set status checks to strict [iceberg-python]

2025-02-10 Thread via GitHub
kevinjqliu commented on code in PR #1640: URL: https://github.com/apache/iceberg-python/pull/1640#discussion_r1949703260 ## .asf.yaml: ## @@ -34,6 +34,10 @@ github: rebase: true protected_branches: main: + required_status_checks: +# strict means "Requir

[PR] Repo: Set status checks to strict [iceberg-python]

2025-02-10 Thread via GitHub
Fokko opened a new pull request, #1640: URL: https://github.com/apache/iceberg-python/pull/1640 We just had an issue with merging a stale branch, which fixed @kevinjqliu right away in https://github.com/apache/iceberg-python/pull/1638 In `iceberg-rust` they have this nice button:

[PR] Break circular import [iceberg-python]

2025-02-10 Thread via GitHub
Fokko opened a new pull request, #1639: URL: https://github.com/apache/iceberg-python/pull/1639 Sometime I'm seeing this: ``` ImportError while loading conftest '/home/runner/work/iceberg-python/iceberg-python/tests/conftest.py'. tests/conftest.py:52: in from pyiceberg.c

  1   2   3   >