Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-11 Thread via GitHub
nastra commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1879880071 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -65,6 +85,25 @@ case class ResolveViews(spark: SparkSess

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-12-11 Thread via GitHub
nastra commented on code in PR #11317: URL: https://github.com/apache/iceberg/pull/11317#discussion_r1856456560 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880074762 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -65,6 +85,25 @@ case class ResolveViews(spark: SparkS

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
dramaticlly commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2536868437 > After taking a closer look at the [Java Doc](https://github.com/apache/spark/blob/branch-3.5/sql/core/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala#L224), I found

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-11 Thread via GitHub
jiakai-li commented on PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#issuecomment-2536887038 Thanks very much for the guidance guys @sungwy and @Fokko . Is it ok for me to pick up the delete part as well? I'll update this PR to include both operations if that's ok. Thank

Re: [PR] docker: Build for `arm64` architecture [iceberg]

2024-12-11 Thread via GitHub
amogh-jahagirdar merged PR #11753: URL: https://github.com/apache/iceberg/pull/11753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-11 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2537210180 @zeroshade hoping you can review when you've got time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] docker: The `archive` seems unstable [iceberg-python]

2024-12-11 Thread via GitHub
Fokko commented on PR #1425: URL: https://github.com/apache/iceberg-python/pull/1425#issuecomment-2537217676 Thanks @sungwy 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] docker: The `archive` seems unstable [iceberg-python]

2024-12-11 Thread via GitHub
Fokko merged PR #1425: URL: https://github.com/apache/iceberg-python/pull/1425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1881029096 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be repr

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1881029096 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be repr

Re: [PR] Update StrictProjection tests [iceberg-python]

2024-12-11 Thread via GitHub
sungwy commented on code in PR #1422: URL: https://github.com/apache/iceberg-python/pull/1422#discussion_r1881033971 ## tests/test_transforms.py: ## @@ -988,608 +997,367 @@ def _test_projection(lhs: Optional[UnboundPredicate[L]], rhs: Optional[UnboundPr raise ValueErro

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
kazuyukitanimura commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2537255407 Just to add @huaxingao's point tableExists(dbName: String, tableName: String): Boolean it is meant to be only for the hardcoded spark_catalog only. But looks like

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
sunny1154 commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2537250201 thanks @huaxingao for looking into this. is `spark.sessionState.catalog.getTableMetadata(TableIdentifier(table, Some(database)))` also expected to work with HMS? currently

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
kazuyukitanimura commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2537281522 @sunny1154 I think you would need to specify the catalog in `TableIdentifier()` Otherwise, Spark tries to use `spark_catalog` https://github.com/apache/spark/blo

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-11 Thread via GitHub
zeroshade commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2537284504 I should be able to give this a review tomorrow or Friday. In the meantime can you resolve the conflict in the go.mod? Thanks! -- This is an automated message from the Apache Git Ser

[I] Drop behavioral change for Spark with REST Catalogs [iceberg]

2024-12-11 Thread via GitHub
c-thiel opened a new issue, #11754: URL: https://github.com/apache/iceberg/issues/11754 ### Feature Request / Improvement Currently when purge-dropping tables with Spark and the REST Catalog, Spark deletes all files of the tables before sending the drop request to the REST Catalog. I

[I] Decouple building and serialization [iceberg-rust]

2024-12-11 Thread via GitHub
Sl1mb0 opened a new issue, #778: URL: https://github.com/apache/iceberg-rust/issues/778 At the moment, the building and serialization of Iceberg metadata is coupled together. For example, let's say I want to build a `ManifestFile` that I then add to a `ManifestList`: (some cod

Re: [I] Support more complex types when reading into arrow record batch. [iceberg-rust]

2024-12-11 Thread via GitHub
sdd commented on issue #405: URL: https://github.com/apache/iceberg-rust/issues/405#issuecomment-2536941270 Hi @ryzhyk - I implemented the default value handling and type promotion limitations mentioned in @liurenjie1024's [comment at the top of the issue](https://github.com/apache/iceberg-

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-12-11 Thread via GitHub
wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1880814678 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [I] Support more complex types when reading into arrow record batch. [iceberg-rust]

2024-12-11 Thread via GitHub
ryzhyk commented on issue #405: URL: https://github.com/apache/iceberg-rust/issues/405#issuecomment-2536958987 Thanks for the update @sdd ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-11 Thread via GitHub
sungwy commented on PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#issuecomment-2536972352 Yes, I think updating this PR to include the changes for both makes sense @jiakai-li 👍 Thank you again for tackling this issue! -- This is an automated message from the Ap

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase/TestRewritePositionDeleteFilesAction [iceberg]

2024-12-11 Thread via GitHub
szehon-ho commented on code in PR #11748: URL: https://github.com/apache/iceberg/pull/11748#discussion_r1880821876 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -275,7 +275,7 @@ public void testRewriteFilter()

Re: [PR] Add `all_manifests` metadata table with tests [iceberg-python]

2024-12-11 Thread via GitHub
soumya-ghosh commented on PR #1241: URL: https://github.com/apache/iceberg-python/pull/1241#issuecomment-2536965547 @Fokko bumping this up for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [WIP][Core] Restrict adding column of StructType with Empty Fields [iceberg]

2024-12-11 Thread via GitHub
singhpk234 opened a new pull request, #11755: URL: https://github.com/apache/iceberg/pull/11755 ## About the change Recently stumbled on a schema where a column was of struct type but the underlying struct was empty, this lead to failure when writing the parquet file because :

Re: [PR] Ignore partition fields that reference a dropped source-id [iceberg-python]

2024-12-11 Thread via GitHub
Fokko closed pull request #1393: Ignore partition fields that reference a dropped source-id URL: https://github.com/apache/iceberg-python/pull/1393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Ignore partition fields that reference a dropped source-id [iceberg-python]

2024-12-11 Thread via GitHub
Fokko commented on PR #1393: URL: https://github.com/apache/iceberg-python/pull/1393#issuecomment-2537019109 This is actually dangerous in the case of V1 tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Added support for lowercase FileFormat for Issue #1340 [iceberg-python]

2024-12-11 Thread via GitHub
Fokko commented on PR #1362: URL: https://github.com/apache/iceberg-python/pull/1362#issuecomment-2537057846 @hgollakota It looks like there is a formatting issue, could you run `make lint`? :) -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] REST: AuthManager API [iceberg]

2024-12-11 Thread via GitHub
nastra commented on PR #10753: URL: https://github.com/apache/iceberg/pull/10753#issuecomment-2535628597 @jbonofre I've talked offline with @adutra and he's going to split up this PR into multiple smaller ones. The diff inside this PR is just too large to review and touches a lot of critica

Re: [I] Discussion: make DataFile Serializable && Deserializable [iceberg-rust]

2024-12-11 Thread via GitHub
Fokko commented on issue #774: URL: https://github.com/apache/iceberg-rust/issues/774#issuecomment-2535718431 Hey @ZENOTME thanks for raising this. Technically the `Datafile` is already serializable, you can encode it into Iceberg Avro :) I know how this works in Java and Python, but

Re: [I] Flink Use distribution-mode: RANGE , null partition bucket will case error [iceberg]

2024-12-11 Thread via GitHub
Guosmilesmile closed issue #11644: Flink Use distribution-mode: RANGE , null partition bucket will case error URL: https://github.com/apache/iceberg/issues/11644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on PR #11317: URL: https://github.com/apache/iceberg/pull/11317#issuecomment-2536685955 https://docs.google.com/document/d/1iPGVCIcr-M0XtAiudOguWAvmqIdVgpYN5vz5ohO8PKw/edit?tab=t.0#heading=h.cr6o1g2rn5hc -- This is an automated message from the Apache Git Service.

Re: [PR] Core, Flink, Spark, KafkaConnect: Remove remaining usage of deprecated path API [iceberg]

2024-12-11 Thread via GitHub
nastra commented on code in PR #11744: URL: https://github.com/apache/iceberg/pull/11744#discussion_r1879895668 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -867,18 +867,17 @@ private List deleteFiles(Table t

[I] Dectect schema evolution or partition evolution for append DataFile [iceberg-rust]

2024-12-11 Thread via GitHub
ZENOTME opened a new issue, #777: URL: https://github.com/apache/iceberg-rust/issues/777 After #349, we support appending DataFile now. But I found there are some check may miss now: When we append DataFile, schema evolution or partition evolution may happen in the table after we generate t

Re: [PR] feat!: Remove `BoundPartitionSpec` [iceberg-rust]

2024-12-11 Thread via GitHub
Fokko commented on PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#issuecomment-2535647234 Gentle ping @liurenjie1024 so we can wrap up the 0.4.0 milestone -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add new standards for deprecating APIs [iceberg-python]

2024-12-11 Thread via GitHub
ndrluis commented on code in PR #1365: URL: https://github.com/apache/iceberg-python/pull/1365#discussion_r1880601202 ## pyiceberg/catalog/__init__.py: ## @@ -631,9 +631,9 @@ def drop_view(self, identifier: Union[str, Identifier]) -> None: """ @deprecated( -

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880067524 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws N

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880072330 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws N

Re: [PR] Core: Assign fresh IDs to view schema [iceberg]

2024-12-11 Thread via GitHub
nastra closed pull request #10253: Core: Assign fresh IDs to view schema URL: https://github.com/apache/iceberg/pull/10253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[PR] Core: Assign fresh IDs to view schema [iceberg]

2024-12-11 Thread via GitHub
nastra opened a new pull request, #10253: URL: https://github.com/apache/iceberg/pull/10253 This addresses https://github.com/apache/iceberg/pull/9596#discussion_r1473187177 and assigns fresh IDs to a View's schema when creating/replacing a view -- This is an automated message from the A

Re: [PR] Spark: Avoid client-side delete of metadata file when purging table [iceberg]

2024-12-11 Thread via GitHub
nastra closed pull request #11752: Spark: Avoid client-side delete of metadata file when purging table URL: https://github.com/apache/iceberg/pull/11752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group by ordinals" [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880074762 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -65,6 +85,25 @@ case class ResolveViews(spark: SparkS

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group by ordinals" [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880074762 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -65,6 +85,25 @@ case class ResolveViews(spark: SparkS

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group by ordinals" [iceberg]

2024-12-11 Thread via GitHub
nastra commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880139155 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws NoSu

[PR] Hive: Add Hive 4 support and remove Hive 3 [iceberg]

2024-12-11 Thread via GitHub
manuzhang opened a new pull request, #11750: URL: https://github.com/apache/iceberg/pull/11750 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] REST: AuthManager API [iceberg]

2024-12-11 Thread via GitHub
jbonofre commented on PR #10753: URL: https://github.com/apache/iceberg/pull/10753#issuecomment-2535919607 @nastra thanks ! much appreciated ! If I understand the request to split in smaller PRs, I'm surprised that it comes now (this PR has been created in July, and several reviews since th

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on PR #11751: URL: https://github.com/apache/iceberg/pull/11751#issuecomment-2536045776 @rdblue hello,Could you please help approve workflow runs and assign a Reviewer? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Tracking issues of iceberg rust v0.4.0 Release [iceberg-rust]

2024-12-11 Thread via GitHub
sungwy commented on issue #739: URL: https://github.com/apache/iceberg-rust/issues/739#issuecomment-2536048804 Also update on https://github.com/apache/iceberg-rust/issues/706 : I've investigated it and it looks like a non-issue. I'm waiting on the issue reporters to respond before closing

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group by ordinals" [iceberg]

2024-12-11 Thread via GitHub
Ppei-Wang commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1880271789 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws N

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-11 Thread via GitHub
sungwy commented on PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#issuecomment-2536123513 Hi @jiakai-li - thank you for reporting this issue and putting up this PR! The behavior looks right to me, but I think it would be worth thinking through this behavior change

Re: [I] Compatibility Issue with pydantic and annotated-types in pyiceberg 0.8.1 [iceberg-python]

2024-12-11 Thread via GitHub
djouallah commented on issue #1418: URL: https://github.com/apache/iceberg-python/issues/1418#issuecomment-2536151795 > @djouallah do you know which version of Pydantic you're using? You can easily check it using: > > ``` > Python 3.10.14 (main, Mar 19 2024, 21:46:16) [Clang 15.0.

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-11 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1880327776 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] docker: Build for `arm64` architecture [iceberg]

2024-12-11 Thread via GitHub
Fokko commented on code in PR #11753: URL: https://github.com/apache/iceberg/pull/11753#discussion_r1880379985 ## .github/workflows/publish-iceberg-rest-fixture-docker.yml: ## @@ -34,23 +34,29 @@ jobs: build: runs-on: ubuntu-latest steps: -- uses: actions/checko

[PR] docker: Build for `arm64` architecture [iceberg]

2024-12-11 Thread via GitHub
Fokko opened a new pull request, #11753: URL: https://github.com/apache/iceberg/pull/11753 We should also publish the `arm64` arch next to `amd64`. I bumped into this when doing some checks on Iceberg-rust. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-11 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1880327776 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] Spark: Avoid client-side delete of metadata file when purging table [iceberg]

2024-12-11 Thread via GitHub
nastra commented on code in PR #11752: URL: https://github.com/apache/iceberg/pull/11752#discussion_r1880395932 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -361,32 +359,18 @@ public boolean purgeTable(Identifier ident) { ValidationE

Re: [PR] Add new standards for deprecating APIs [iceberg-python]

2024-12-11 Thread via GitHub
ndrluis commented on PR #1365: URL: https://github.com/apache/iceberg-python/pull/1365#issuecomment-2536298542 Some Deprecation Examples Future Deprecation https://github.com/user-attachments/assets/cfa68721-3a60-427b-94f2-36eb346b6825"; /> Deprecated https://github.com/u

Re: [PR] Add new standards for deprecating APIs [iceberg-python]

2024-12-11 Thread via GitHub
Fokko commented on code in PR #1365: URL: https://github.com/apache/iceberg-python/pull/1365#discussion_r1880418065 ## pyiceberg/catalog/__init__.py: ## @@ -631,9 +631,9 @@ def drop_view(self, identifier: Union[str, Identifier]) -> None: """ @deprecated( -

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-11 Thread via GitHub
Fokko commented on PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#issuecomment-2536336980 First of all, thanks @jiakai-li for raising this PR, and thanks @sungwy for raising that issue. And I agree, I think we should also be able to control case-sensitivity when doing del

Re: [PR] docs: Use `load_catalog` instead [iceberg-python]

2024-12-11 Thread via GitHub
Fokko merged PR #1406: URL: https://github.com/apache/iceberg-python/pull/1406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] docker: Build for `arm64` architecture [iceberg]

2024-12-11 Thread via GitHub
Fokko commented on code in PR #11753: URL: https://github.com/apache/iceberg/pull/11753#discussion_r1880434138 ## .github/workflows/publish-iceberg-rest-fixture-docker.yml: ## @@ -34,23 +34,29 @@ jobs: build: runs-on: ubuntu-latest steps: -- uses: actions/checko

Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]

2024-12-11 Thread via GitHub
dongjoon-hyun commented on PR #11731: URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2536419878 Thank you for checking Apache Iceberg side, @pan3793 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Support more complex types when reading into arrow record batch. [iceberg-rust]

2024-12-11 Thread via GitHub
ryzhyk commented on issue #405: URL: https://github.com/apache/iceberg-rust/issues/405#issuecomment-2536430104 @sdd, I'm curious about the status of this. I ran into this issue today and noticed that you're working on a fix, so was wondering if you still plan to do it. -- This is an auto

Re: [PR] Core: Merge conflicting deletion vectors [iceberg]

2024-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #11693: URL: https://github.com/apache/iceberg/pull/11693#discussion_r1880489562 ## core/src/main/java/org/apache/iceberg/deletes/DVFileWriter.java: ## @@ -36,6 +36,22 @@ public interface DVFileWriter extends Closeable { */ void de

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on PR #11317: URL: https://github.com/apache/iceberg/pull/11317#issuecomment-2536523984 Bringing this up in Community Sync today to discuss future of the API here -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase/TestRewritePositionDeleteFilesAction [iceberg]

2024-12-11 Thread via GitHub
huaxingao commented on PR #11748: URL: https://github.com/apache/iceberg/pull/11748#issuecomment-2536559206 cc @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
huaxingao commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2536571605 @dramaticlly Thanks for pinging me. This seems to be a Spark bug. I'll investigate further. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Core, Flink, Spark, KafkaConnect: Remove remaining usage of deprecated path API [iceberg]

2024-12-11 Thread via GitHub
amogh-jahagirdar commented on PR #11744: URL: https://github.com/apache/iceberg/pull/11744#issuecomment-2536572166 Thanks for reviewing @nastra @ebyhr! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]

2024-12-11 Thread via GitHub
flyrain commented on issue #11741: URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2536580015 Yeah, it's more like a Spark bug, probably the Iceberg REST catalog didn't impl this method in class `org.apache.spark.sql.catalog.Catalog` well. ``` public abstract boolean

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-12-11 Thread via GitHub
twuebi commented on PR #11317: URL: https://github.com/apache/iceberg/pull/11317#issuecomment-2536586079 Thanks for bringing it there. Where can I find the calendar for today's Community Sync? I've been looking through the iceberg community page but there's only a calendar having the

Re: [PR] Core, Flink, Spark, KafkaConnect: Remove remaining usage of deprecated path API [iceberg]

2024-12-11 Thread via GitHub
amogh-jahagirdar merged PR #11744: URL: https://github.com/apache/iceberg/pull/11744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Distributed writes in the same iceberg transaction [iceberg-python]

2024-12-11 Thread via GitHub
jimmyxie-figma commented on issue #357: URL: https://github.com/apache/iceberg-python/issues/357#issuecomment-2537304523 Any update on supporting distributed write, we are also interested in adding iceberg write capability to Ray. https://github.com/ray-project/ray/issues/49032 -- Thi

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881105008 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881105376 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881105847 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881106567 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881103731 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881104193 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881106923 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1881100218 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be repr

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-12-11 Thread via GitHub
wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1881085717 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-11 Thread via GitHub
ajreid21 commented on PR #11756: URL: https://github.com/apache/iceberg/pull/11756#issuecomment-2537316670 @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Spec: Document Snapshot Summary Optional Fields for Standardization [iceberg]

2024-12-11 Thread via GitHub
RussellSpitzer commented on code in PR #11660: URL: https://github.com/apache/iceberg/pull/11660#discussion_r1881103356 ## format/spec.md: ## @@ -693,6 +686,64 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is t

Re: [PR] support uri not end with character '/' in rest catalog [iceberg]

2024-12-11 Thread via GitHub
beyond-up commented on PR #11392: URL: https://github.com/apache/iceberg/pull/11392#issuecomment-2534925339 > This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull reques

Re: [PR] Add clang format [iceberg-cpp]

2024-12-11 Thread via GitHub
gaborkaszab commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1879660010 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +#

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase [iceberg]

2024-12-11 Thread via GitHub
ebyhr commented on PR #11748: URL: https://github.com/apache/iceberg/pull/11748#issuecomment-2535253342 Can we fix `TestRewritePositionDeleteFilesAction` as well? https://github.com/apache/iceberg/blob/ff813445916bfd6ec1cc30a02b02f8bade7a26f6/spark/v3.5/spark/src/test/java/org/apache

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase/TestRewritePositionDeleteFilesAction [iceberg]

2024-12-11 Thread via GitHub
wzx140 commented on PR #11748: URL: https://github.com/apache/iceberg/pull/11748#issuecomment-2535270455 > Can we fix `TestRewritePositionDeleteFilesAction` as well? > > https://github.com/apache/iceberg/blob/ff813445916bfd6ec1cc30a02b02f8bade7a26f6/spark/v3.5/spark/src/test/java/org/

Re: [PR] Core, Spark3.5: Fix tests failure due to timeout [iceberg]

2024-12-11 Thread via GitHub
manuzhang commented on PR #11654: URL: https://github.com/apache/iceberg/pull/11654#issuecomment-2535495990 @nastra Please take another look, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

2024-12-11 Thread via GitHub
jbonofre commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2534627053 Should we reopen this PR or is it superseded by another one ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] feat: Implement Decimal from/to bytes represents [iceberg-rust]

2024-12-11 Thread via GitHub
Xuanwo commented on code in PR #665: URL: https://github.com/apache/iceberg-rust/pull/665#discussion_r1879569854 ## crates/iceberg/src/spec/values.rs: ## @@ -1012,6 +1053,33 @@ impl Datum { } } +/// Try to create a decimal literal from [`Decimal`] with precis

Re: [PR] doc: add RisingWave to users [iceberg-rust]

2024-12-11 Thread via GitHub
Xuanwo merged PR #775: URL: https://github.com/apache/iceberg-rust/pull/775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-11 Thread via GitHub
pitrou commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1879713332 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# di

Re: [PR] fix: return type for year and month transform should be int [iceberg-rust]

2024-12-11 Thread via GitHub
xxchan commented on PR #776: URL: https://github.com/apache/iceberg-rust/pull/776#issuecomment-2535384354 cc @sdd @Xuanwo @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2024-12-11 Thread via GitHub
petehanssens commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2535436550 Super keen to see this happen too! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-11 Thread via GitHub
pitrou commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1879619335 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# di

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-11 Thread via GitHub
gaborkaszab commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1879710621 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

[PR] doc: add note for `day` transform [iceberg]

2024-12-11 Thread via GitHub
xxchan opened a new pull request, #11749: URL: https://github.com/apache/iceberg/pull/11749 This was very confusing related: https://github.com/apache/iceberg-rust/issues/478, https://github.com/apache/iceberg/issues/10616 cc @Fokko @sdd -- This is an automated message from

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-12-11 Thread via GitHub
jbonofre commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2535341659 @danielcweeks @RussellSpitzer @aokolnychyi would you have some time to take a look on this PR and my proposal (previous comment) ? -- This is an automated message from the Apache Git

Re: [PR] REST: AuthManager API [iceberg]

2024-12-11 Thread via GitHub
jbonofre commented on PR #10753: URL: https://github.com/apache/iceberg/pull/10753#issuecomment-2535347083 @nastra @danielcweeks @RussellSpitzer I thought we had a consensus about moving forward on this PR, targeting 1.8.0. Can we move forward on this one ? Thanks ! -- This is an automat

Re: [PR] Spec: add variant type [iceberg]

2024-12-11 Thread via GitHub
XBaith commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1879742066 ## format/spec.md: ## @@ -1208,6 +1224,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`struct`** | `group`

  1   2   >