Re: [PR] Spec additions for encryption [iceberg]

2025-02-12 Thread via GitHub
ggershinsky commented on PR #12162: URL: https://github.com/apache/iceberg/pull/12162#issuecomment-2655788945 thanks @rdblue and @RussellSpitzer for the productive call and the suggestions. I've updated the PR to reflect the discussion results. All comments and corrections are welcome. I

Re: [PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-12 Thread via GitHub
nastra commented on code in PR #12242: URL: https://github.com/apache/iceberg/pull/12242#discussion_r1953981755 ## site/docs/releases.md: ## @@ -72,6 +72,77 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.8.0 release + +Apache Ic

Re: [PR] Docs: Add 1.8.0 versioned docs [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar merged PR #12243: URL: https://github.com/apache/iceberg/pull/12243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] Update release version to 1.8.0 in doap.rdf [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12247: URL: https://github.com/apache/iceberg/pull/12247 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Docs: Add 1.8.0 versioned docs [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar commented on PR #12243: URL: https://github.com/apache/iceberg/pull/12243#issuecomment-2655732007 Thanks for reviewing @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] TestS3FileIO fails locally [iceberg]

2025-02-12 Thread via GitHub
nastra commented on issue #12237: URL: https://github.com/apache/iceberg/issues/12237#issuecomment-2655725383 I've talked to @ebyhr on Slack and he's using OrbStack while I'm using Docker Desktop. After switching to OrbStack locally the tests started to pass again. So we need to figure out

Re: [PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #12242: URL: https://github.com/apache/iceberg/pull/12242#discussion_r1953935634 ## site/docs/releases.md: ## @@ -72,6 +72,77 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.8.0 release +

Re: [PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-12 Thread via GitHub
nastra commented on code in PR #12242: URL: https://github.com/apache/iceberg/pull/12242#discussion_r1953929532 ## site/docs/releases.md: ## @@ -72,6 +72,77 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.8.0 release + +Apache Ic

[PR] Docs: Add 1.8.0 javadocs [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12246: URL: https://github.com/apache/iceberg/pull/12246 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Docs: Add 1.8.0 Javadocs [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar closed pull request #12245: Docs: Add 1.8.0 Javadocs URL: https://github.com/apache/iceberg/pull/12245 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[PR] Docs: Add 1.8.0 Javadocs [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12245: URL: https://github.com/apache/iceberg/pull/12245 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953883726 ## format/spec.md: ## @@ -392,7 +392,7 @@ In v3 and later, an Iceberg table can track row lineage fields for all newly cre These fields are assigned and updated by

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1953886162 ## tests/catalog/test_sql.py: ## @@ -1116,17 +1116,30 @@ def test_create_namespace_with_empty_identifier(catalog: SqlCatalog, empty_names lazy_fixtu

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953883726 ## format/spec.md: ## @@ -392,7 +392,7 @@ In v3 and later, an Iceberg table can track row lineage fields for all newly cre These fields are assigned and updated by

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953883726 ## format/spec.md: ## @@ -392,7 +392,7 @@ In v3 and later, an Iceberg table can track row lineage fields for all newly cre These fields are assigned and updated by

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953883726 ## format/spec.md: ## @@ -392,7 +392,7 @@ In v3 and later, an Iceberg table can track row lineage fields for all newly cre These fields are assigned and updated by

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953883726 ## format/spec.md: ## @@ -392,7 +392,7 @@ In v3 and later, an Iceberg table can track row lineage fields for all newly cre These fields are assigned and updated by

Re: [I] java.lang.ClassNotFoundException: .package [iceberg]

2025-02-12 Thread via GitHub
melin closed issue #12231: java.lang.ClassNotFoundException: .package URL: https://github.com/apache/iceberg/issues/12231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] feat: support position delete writer [iceberg-rust]

2025-02-12 Thread via GitHub
jonathanc-n commented on code in PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#discussion_r1953805959 ## crates/iceberg/src/writer/base_writer/position_delete_file_writer.rs: ## @@ -0,0 +1,245 @@ +// Licensed to the Apache Software Foundation (ASF) under one +//

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-12 Thread via GitHub
github-actions[bot] commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1953813017 ## bindings/c/.gitignore: ## @@ -0,0 +1 @@ +build/ Review Comment: ```suggestion # Licensed to the Apache Software Foundation (ASF) under one #

Re: [PR] feat: support position delete writer [iceberg-rust]

2025-02-12 Thread via GitHub
jonathanc-n commented on code in PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#discussion_r1953805959 ## crates/iceberg/src/writer/base_writer/position_delete_file_writer.rs: ## @@ -0,0 +1,245 @@ +// Licensed to the Apache Software Foundation (ASF) under one +//

Re: [I] TestS3FileIO fails locally [iceberg]

2025-02-12 Thread via GitHub
ebyhr commented on issue #12237: URL: https://github.com/apache/iceberg/issues/12237#issuecomment-2655481087 All tests passed on my laptop. I tested with the following environment: ``` Gradle 8.12.1 -

[PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-12 Thread via GitHub
Xuanwo opened a new pull request, #966: URL: https://github.com/apache/iceberg-rust/pull/966 This PR introduces a C FFI for iceberg-rust, allowing users to interact with Iceberg from C. This PR is a PoC demonstrating how this can be implemented and what it will look like. Most of the

Re: [I] How to add 2 different catalog-type in spark config [iceberg]

2025-02-12 Thread via GitHub
AwasthiSomesh commented on issue #12234: URL: https://github.com/apache/iceberg/issues/12234#issuecomment-2655473817 @RussellSpitzer Thanks for your update let me try this and update you if its working as expected. Thanks, Somesh -- This is an automated message from the Apache

[PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12242: URL: https://github.com/apache/iceberg/pull/12242 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-12 Thread via GitHub
gabeiglio commented on PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#issuecomment-2655372996 Thanks for the help! @kevinjqliu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump cryptography from 43.0.3 to 44.0.1 [iceberg-python]

2025-02-12 Thread via GitHub
dependabot[bot] closed pull request #1659: Build: Bump cryptography from 43.0.3 to 44.0.1 URL: https://github.com/apache/iceberg-python/pull/1659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-12 Thread via GitHub
jonathanc-n commented on PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#issuecomment-2655357114 Tests have been added, this implementation follows the pyiceberg implementation. cc @Fokko @liurenjie1024 @ZENOTME -- This is an automated message from the Apache Git Service. T

Re: [PR] Build: Bump cryptography from 43.0.3 to 44.0.1 [iceberg-python]

2025-02-12 Thread via GitHub
dependabot[bot] commented on PR #1659: URL: https://github.com/apache/iceberg-python/pull/1659#issuecomment-2655348154 Looks like cryptography is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] [feat] push down filters and positional deletes to the record batch level [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu closed issue #1050: [feat] push down filters and positional deletes to the record batch level URL: https://github.com/apache/iceberg-python/issues/1050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] [feat] push down filters and positional deletes to the record batch level [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu closed issue #1050: [feat] push down filters and positional deletes to the record batch level URL: https://github.com/apache/iceberg-python/issues/1050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu merged PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Metadata location fallback [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1658: URL: https://github.com/apache/iceberg-python/pull/1658#issuecomment-2655346808 hey @IsaacWarren what operations were you performing when you encountered the issue above? was it a create_table or load_table or something else? > don't populate tab

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2655333694 > looks like there are some changes that we previous removed that are now back :/ > > like changes in `pyproject.toml` > > > > @mattmartin14 do you mind

[PR] Build: Bump cryptography from 43.0.3 to 44.0.1 [iceberg-python]

2025-02-12 Thread via GitHub
dependabot[bot] opened a new pull request, #1659: URL: https://github.com/apache/iceberg-python/pull/1659 Bumps [cryptography](https://github.com/pyca/cryptography) from 43.0.3 to 44.0.1. Changelog Sourced from https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst";>cryptogr

Re: [PR] Added support for Polars DataFrame and LazyFrame [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu merged PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Added support for Polars DataFrame and LazyFrame [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2655284900 LGTM! Thanks for the contribution @yigal-rozenberg I merged main to fix the conflicts -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953671095 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,77 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requ

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1953646235 ## poetry.lock: ## @@ -1,13 +1,11 @@ -# This file is automatically @generated by Poetry 2.0.1 and should not be changed by hand. +# This file is automaticall

Re: [PR] [infra] update poetry if necessary [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu merged PR #1656: URL: https://github.com/apache/iceberg-python/pull/1656 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Build: Bump mkdocs-material from 9.6.3 to 9.6.4 [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu merged PR #1657: URL: https://github.com/apache/iceberg-python/pull/1657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2655235254 looks like there are some changes that we previous removed that are now back :/ like changes in `pyproject.toml` @mattmartin14 do you mind if i push these commits to a

Re: [PR] Docs: Add JavaDocs for 1.8.0 release [iceberg]

2025-02-12 Thread via GitHub
amogh-jahagirdar closed pull request #12241: Docs: Add JavaDocs for 1.8.0 release URL: https://github.com/apache/iceberg/pull/12241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953621390 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,77 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953620434 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,77 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-12 Thread via GitHub
dramaticlly commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1953619836 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics o

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953612580 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -81,6 +82,15 @@ public static PrimitiveType fromPrimitiveString(String typeString) { throw new Il

[PR] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

2025-02-12 Thread via GitHub
sririshindra opened a new pull request, #12240: URL: https://github.com/apache/iceberg/pull/12240 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-12 Thread via GitHub
dramaticlly commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1953617634 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics o

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953613226 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -788,6 +792,9 @@ public static T visit(Type type, CustomOrderSchemaVisitor visitor) {

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953616108 ## api/src/test/java/org/apache/iceberg/types/TestTypes.java: ## @@ -43,6 +67,13 @@ public void fromPrimitiveString() { assertThat(Types.fromPrimitiveString("D

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953614605 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,77 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953613423 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,77 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: add variant type support [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1953612378 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -712,6 +712,10 @@ public T map(Types.MapType map, Supplier keyResult, Supplier valueResult)

Re: [PR] OpenAPI: Add overwrite option when registering an iceberg table [iceberg]

2025-02-12 Thread via GitHub
dramaticlly commented on code in PR #12239: URL: https://github.com/apache/iceberg/pull/12239#discussion_r1953605674 ## open-api/rest-catalog-open-api.py: ## @@ -502,6 +502,9 @@ class PlanStatus(BaseModel): class RegisterTableRequest(BaseModel): name: str metadata_loc

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-12 Thread via GitHub
rdblue commented on code in PR #12060: URL: https://github.com/apache/iceberg/pull/12060#discussion_r1953603432 ## core/src/main/java/org/apache/iceberg/V1Metadata.java: ## @@ -236,34 +235,38 @@ static Types.StructType dataFileSchema(Types.StructType partitionType) { }

Re: [I] A move after a rename fails [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on issue #10830: URL: https://github.com/apache/iceberg/issues/10830#issuecomment-2655136097 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] OpenAPI: Add overwrite option when registering an iceberg table [iceberg]

2025-02-12 Thread via GitHub
stevenzwu commented on code in PR #12239: URL: https://github.com/apache/iceberg/pull/12239#discussion_r1953584278 ## open-api/rest-catalog-open-api.py: ## @@ -502,6 +502,9 @@ class PlanStatus(BaseModel): class RegisterTableRequest(BaseModel): name: str metadata_locat

Re: [I] A move after a rename fails [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] closed issue #10830: A move after a rename fails URL: https://github.com/apache/iceberg/issues/10830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Running MERGE INTO with more than one WHEN condition fails if the number of columns in the target table is > 321 [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] closed issue #10294: Running MERGE INTO with more than one WHEN condition fails if the number of columns in the target table is > 321 URL: https://github.com/apache/iceberg/issues/10294 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Avro: Add variant type support [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on PR #11934: URL: https://github.com/apache/iceberg/pull/11934#issuecomment-2655136314 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Support Conditional Transaction Commits [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] closed issue #6514: Support Conditional Transaction Commits URL: https://github.com/apache/iceberg/issues/6514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Remove unneeded metadata read during update event generation [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on PR #11829: URL: https://github.com/apache/iceberg/pull/11829#issuecomment-2655136284 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core, Spark: Minimize executor memory pressure in broadcast of data to delete files [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on PR #11957: URL: https://github.com/apache/iceberg/pull/11957#issuecomment-2655136340 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] support uri not end with character '/' in rest catalog [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on PR #11392: URL: https://github.com/apache/iceberg/pull/11392#issuecomment-2655136210 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] The "Emitting watermarks" feature can't be used in flink sql? [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] closed issue #10219: The "Emitting watermarks" feature can't be used in flink sql? URL: https://github.com/apache/iceberg/issues/10219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Support Conditional Transaction Commits [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on issue #6514: URL: https://github.com/apache/iceberg/issues/6514#issuecomment-2655135853 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] The "Emitting watermarks" feature can't be used in flink sql? [iceberg]

2025-02-12 Thread via GitHub
github-actions[bot] commented on issue #10219: URL: https://github.com/apache/iceberg/issues/10219#issuecomment-2655136010 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

[PR] Metadata location fallback [iceberg-python]

2025-02-12 Thread via GitHub
IsaacWarren opened a new pull request, #1658: URL: https://github.com/apache/iceberg-python/pull/1658 Some REST catalogs (Polaris when I was testing an Azure catalog) don't populate table_response.metadata_location but they do populate table_response.metadata.location. This PR just falls ba

[PR] OpenAPI: Add overwrite option when registering an iceberg table [iceberg]

2025-02-12 Thread via GitHub
dramaticlly opened a new pull request, #12239: URL: https://github.com/apache/iceberg/pull/12239 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[PR] Build: Bump mkdocs-material from 9.6.3 to 9.6.4 [iceberg-python]

2025-02-12 Thread via GitHub
dependabot[bot] opened a new pull request, #1657: URL: https://github.com/apache/iceberg-python/pull/1657 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.3 to 9.6.4. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mk

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-12 Thread via GitHub
zeroshade commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1953480223 ## catalog/rest/rest.go: ## @@ -989,3 +992,95 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, nil

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2654988421 i've tried something a little different to resolve the lock file conflict; i'll see if it works; added my company's repo with the option of priority==explicit; it looks like i

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-12 Thread via GitHub
zeroshade commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1953483382 ## catalog/rest/rest_test.go: ## @@ -1330,6 +1331,252 @@ func (r *RestCatalogSuite) TestRegisterTable409() { r.ErrorContains(err, "The given table already ex

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-12 Thread via GitHub
zeroshade commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1953482371 ## catalog/rest/rest_test.go: ## @@ -1330,6 +1331,252 @@ func (r *RestCatalogSuite) TestRegisterTable409() { r.ErrorContains(err, "The given table already ex

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2654939057 @Fokko @kevinjqliu - for context on this poetry lock file issue that keeps coming up: If i run poetry.lock locally, it wil embed my company's private repo throughout t

Re: [PR] Implement Kerberos authentication support for Hive Catalog [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu closed pull request #766: Implement Kerberos authentication support for Hive Catalog URL: https://github.com/apache/iceberg-python/pull/766 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] API: Follow up on adding Variant data type to implement sanitizing for Variant [iceberg]

2025-02-12 Thread via GitHub
aihuaxu commented on issue #11479: URL: https://github.com/apache/iceberg/issues/11479#issuecomment-2654905064 @ajayrajsaini I think it's not fixed yet. Feel free to work on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-12 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1953427985 ## core/src/test/java/org/apache/iceberg/TestTables.java: ## @@ -93,6 +93,26 @@ public static TestTable create( return new TestTable(ops, name, reporter);

Re: [PR] Support reading initial-defaults [iceberg-python]

2025-02-12 Thread via GitHub
gabeiglio commented on PR #1644: URL: https://github.com/apache/iceberg-python/pull/1644#issuecomment-2654880947 Since initial-default projection happens after filtering in _task_to_record_batches wondering if this will yield the correct results given a pyarrow_filter for this field. --

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2654870836 @Fokko , @kevinjqliu - i think we are good on teh vendor files; i just need your help one more time computing the poetry.lock file, since if i do it on my end, it will embed

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1953415874 ## vendor/hive_metastore/ThriftHiveMetastore.py: ## @@ -72957,4 +72957,4 @@ def __ne__(self, other): ), # 1 ) fix_spec(all_structs) -del all_structs

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
Fokko commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1953389250 ## vendor/fb303/FacebookService.py: ## @@ -2417,4 +2417,4 @@ def __ne__(self, other): all_structs.append(shutdown_args) shutdown_args.thrift_spec = () fix_spec(

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
RussellSpitzer commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953375995 ## format/spec.md: ## @@ -1766,4 +1766,4 @@ The Geometry and Geography class hierarchy and its Well-known text (WKT) and Wel Points are always defined by t

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-12 Thread via GitHub
geruh commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953384729 ## pyiceberg/catalog/__init__.py: ## @@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.t

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
RussellSpitzer commented on PR #12230: URL: https://github.com/apache/iceberg/pull/12230#issuecomment-2654818937 @stevenzwu , @pvary + @singhpk234 - Wording Updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2025-02-12 Thread via GitHub
nabaskes commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2654798498 @amitgilad3 thank you so much, that was so so fast ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-12 Thread via GitHub
RussellSpitzer commented on code in PR #12230: URL: https://github.com/apache/iceberg/pull/12230#discussion_r1953360394 ## format/spec.md: ## @@ -1766,4 +1766,4 @@ The Geometry and Geography class hierarchy and its Well-known text (WKT) and Wel Points are always defined by t

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-12 Thread via GitHub
gabeiglio commented on PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#issuecomment-2654787238 - 32 tests in tests_reads.py test filtering using case sensitive, and insensitive, multiple fields, one field, empty fields, nulls, etc. - 4 tests test filtering partitioned ta

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-12 Thread via GitHub
Fokko commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953356977 ## pyiceberg/catalog/__init__.py: ## @@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.t

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2654758986 > ah thats odd. lets do this the manual way > > for these files > > * vendor/fb303/FacebookService.py > * vendor/hive_metastore/ThriftHiveMetastore.py > * ve

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#issuecomment-2654755991 its green! i'll review this so we can include it as the upcoming 0.9.0 release -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-12 Thread via GitHub
aokolnychyi commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1953337243 ## api/src/main/java/org/apache/iceberg/UpdateSchema.java: ## @@ -67,24 +70,52 @@ default UpdateSchema addColumn(String name, Type type) { } /** - * Add

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-12 Thread via GitHub
aokolnychyi commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1953337243 ## api/src/main/java/org/apache/iceberg/UpdateSchema.java: ## @@ -67,24 +70,52 @@ default UpdateSchema addColumn(String name, Type type) { } /** - * Add

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2654749976 ah thats odd. lets do this the manual way for these files * vendor/fb303/FacebookService.py * vendor/hive_metastore/ThriftHiveMetastore.py * vendor/hive_metastore/

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2025-02-12 Thread via GitHub
amitgilad3 commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2654746253 @nabaskes - https://github.com/zinking/iceberg/pull/5/ this is an attempt to finish the pr -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-12 Thread via GitHub
kevinjqliu commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r1953329429 ## catalog/rest/rest.go: ## @@ -989,3 +989,54 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, ni

Re: [PR] [infra] update poetry if necessary [iceberg-python]

2025-02-12 Thread via GitHub
kevinjqliu commented on PR #1656: URL: https://github.com/apache/iceberg-python/pull/1656#issuecomment-2654730482 I think this **will** enforce the right poetry version is used, otherwise the hash for `poetry.lock` will be different -- This is an automated message from the Apache Git Ser

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-12 Thread via GitHub
pvary commented on code in PR #12060: URL: https://github.com/apache/iceberg/pull/12060#discussion_r1953323496 ## core/src/main/java/org/apache/iceberg/InternalData.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contribu

Re: [I] [feature] Table Scan should take into account the table's sort order [iceberg-python]

2025-02-12 Thread via GitHub
Fokko commented on issue #1637: URL: https://github.com/apache/iceberg-python/issues/1637#issuecomment-2654723140 Keep in mind that the sort order is not a global sort-order but on a file level. This is very nice if you do joins (using the [sort-merge strategy](https://en.wikipedia.org/wik

  1   2   3   >