Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2507255659 @mxm I'm sorry, I'm a bit confused and would like to ask for your advice. Which part do I need to modify? Based on the current modifications, simply change TypeSerializerSc

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

mxm commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2507208103 @Guosmilesmile Ideally, we want to return compatibleAfterMigration() and make sure the old and new serializer can be instantiated. -- This is an automated message from the Apache Git Ser

[PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

advancedxy opened a new pull request, #10755: URL: https://github.com/apache/iceberg/pull/10755 This is a continue work of #3462, all the credits should goes to @RussellSpitzer. Previously there was no way to remove partition specs from a table once they were added. To fix this we

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

advancedxy closed pull request #10755: API: Support removeUnusedSpecs in ExpireSnapshots URL: https://github.com/apache/iceberg/pull/10755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

huaxingao commented on PR #11439: URL: https://github.com/apache/iceberg/pull/11439#issuecomment-2507166143 @ludlows Thanks for the quick fix. Can we have a test that fails without the fix but passes with it? It seems that all your current tests pass even without the fix. -- This is an a

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

Syed-bug commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2507125686 @jurossiar I am getting nosuchtable exception now ... Spark version 3.4.1 Scala version:2.12 iceberg-spark-runtime jar version is 3.4.2.12-1.6.1.jar hadoop-aws jar versi

Re: [PR] Spark: 4.0 snapshot support [iceberg]

pan3793 commented on code in PR #11583: URL: https://github.com/apache/iceberg/pull/11583#discussion_r1862876580 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/IcebergCall.scala: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Spark: 4.0 snapshot support [iceberg]

pan3793 commented on code in PR #11583: URL: https://github.com/apache/iceberg/pull/11583#discussion_r1862876580 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/IcebergCall.scala: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2506978587 @mxm @stevenzwu @pvary Thank you all for your suggestions. I have submitted a version that mainly modifies the SortKeySerializerSnapshot and implements version detection. If the v

Re: [PR] Spark: remove ROW_POSITION from project schema [iceberg]

huaxingao commented on PR #11610: URL: https://github.com/apache/iceberg/pull/11610#issuecomment-2506923142 Thanks @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862801792 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] Core: Map methods should return immutable collections [iceberg]

github-actions[bot] commented on PR #11304: URL: https://github.com/apache/iceberg/pull/11304#issuecomment-2506874239 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Iceberg/Comet integration POC [iceberg]

github-actions[bot] commented on PR #9841: URL: https://github.com/apache/iceberg/pull/9841#issuecomment-2506874150 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] name mapping serde [iceberg-rust]

barronw commented on code in PR #740: URL: https://github.com/apache/iceberg-rust/pull/740#discussion_r1862755672 ## crates/iceberg/src/spec/mod.rs: ## @@ -20,6 +20,7 @@ mod datatypes; mod manifest; mod manifest_list; +mod name_mapping; Review Comment: Please let me know

Re: [PR] name mapping serde [iceberg-rust]

barronw commented on code in PR #740: URL: https://github.com/apache/iceberg-rust/pull/740#discussion_r1862755672 ## crates/iceberg/src/spec/mod.rs: ## @@ -20,6 +20,7 @@ mod datatypes; mod manifest; mod manifest_list; +mod name_mapping; Review Comment: Please let me know

Re: [PR] name mapping serde [iceberg-rust]

barronw commented on code in PR #740: URL: https://github.com/apache/iceberg-rust/pull/740#discussion_r1862755672 ## crates/iceberg/src/spec/mod.rs: ## @@ -20,6 +20,7 @@ mod datatypes; mod manifest; mod manifest_list; +mod name_mapping; Review Comment: Please let me know

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

stevenzwu commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862733192 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

stevenzwu commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862736708 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

stevenzwu commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862733192 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] Core,API: Set `503: added_snapshot_id` as required [iceberg]

Fokko commented on PR #11626: URL: https://github.com/apache/iceberg/pull/11626#issuecomment-2506814718 Great suggestion @rdblue, thanks for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core,API: Set `503: added_snapshot_id` as required [iceberg]

Fokko merged PR #11626: URL: https://github.com/apache/iceberg/pull/11626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

flyrain commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1862708194 ## core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java: ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

flyrain commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1862650256 ## core/src/main/java/org/apache/iceberg/TableMetadataUtil.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [PR] Create publish-docker.yml [iceberg]

sungwy commented on PR #11632: URL: https://github.com/apache/iceberg/pull/11632#issuecomment-2506745817 @Fokko @ajantha-bhat and @jbonofre - thank you all for the reviews on the PR! I've verified the publication using both the `tag` triggered and `workflow_dispatch` triggered workflo

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862663385 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862650411 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862650874 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on code in PR #11365: URL: https://github.com/apache/iceberg/pull/11365#discussion_r1862650411 ## format/view-spec.md: ## @@ -97,7 +97,10 @@ Summary is a string to string map of metadata about a view version. Common metad View definitions can be represent

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

Fokko merged PR #11421: URL: https://github.com/apache/iceberg/pull/11421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark: remove ROW_POSITION from project schema [iceberg]

flyrain commented on PR #11610: URL: https://github.com/apache/iceberg/pull/11610#issuecomment-2506685944 Thanks for the change, @huaxingao ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Spark: remove ROW_POSITION from project schema [iceberg]

flyrain merged PR #11610: URL: https://github.com/apache/iceberg/pull/11610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[I] Tracking issues of iceberg rust v0.4.0 Release [iceberg-rust]

sungwy opened a new issue, #739: URL: https://github.com/apache/iceberg-rust/issues/739 This issue is used to track tasks of the iceberg rust 0.4.0 release. ## Tasks ### Blockers > Blockers are the tasks that must be completed before the release. - [ ] https://gith

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

singhpk234 commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1862514439 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterable.java: ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Create publish-docker.yml [iceberg]

sungwy commented on code in PR #11632: URL: https://github.com/apache/iceberg/pull/11632#discussion_r1862600872 ## .github/workflows/publish-docker.yml: ## @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements

Re: [PR] "how to release", add `gh` command to trigger workflow [iceberg-python]

kevinjqliu commented on PR #1387: URL: https://github.com/apache/iceberg-python/pull/1387#issuecomment-2506626087 @ndrluis thats a good point! something like `make release` will be helpful. I've been just making incremental improvements to the release process. -- This is an automated me

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862580698 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundat

Re: [I] Count rows as a metadata-only operation [iceberg-python]

tusharchou commented on issue #1223: URL: https://github.com/apache/iceberg-python/issues/1223#issuecomment-2506625672 Hi @Fokko, Thank you for helping. I attempted to implement `.count()` in `DataScan`. I can test for it using the `SqlCatalog` in `catalog/test_sql` however when I try

Re: [PR] Kevinjqliu/run integration test on GitHub action macos [iceberg-python]

kevinjqliu commented on PR #1193: URL: https://github.com/apache/iceberg-python/pull/1193#issuecomment-2506624255 superseded by #1370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Kevinjqliu/run integration test on GitHub action macos [iceberg-python]

kevinjqliu closed pull request #1193: Kevinjqliu/run integration test on GitHub action macos URL: https://github.com/apache/iceberg-python/pull/1193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Count rows as a metadata only operation [iceberg-python]

kevinjqliu commented on code in PR #1388: URL: https://github.com/apache/iceberg-python/pull/1388#discussion_r1862579074 ## pyiceberg/table/__init__.py: ## @@ -1493,6 +1496,13 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arrow())

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862577329 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/VendedAzureSasCredentialProvider.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862575461 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/VendedAzureSasCredentialProvider.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862575184 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/VendedAzureSasCredentialProvider.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software F

[PR] Count rows as a metadata only operation [iceberg-python]

tusharchou opened a new pull request, #1388: URL: https://github.com/apache/iceberg-python/pull/1388 closes issue: Count rows as a metadata-only operation #1223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862574536 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

ChaladiMohanVamsi commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1862573721 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

singhpk234 commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1862516655 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterable.java: ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862492365 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862492365 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

pvary commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2506498681 And here is the relevant mailing list thread: https://lists.apache.org/thread/knl1ol7s1o2p7rglgl2mm8c5mc2pk0sx @ajantha-bhat: Are you still working on the proposal? -- This is an

Re: [I] java.lang.IllegalStateException: Connection pool shut down in Spark [iceberg]

singhpk234 commented on issue #11633: URL: https://github.com/apache/iceberg/issues/11633#issuecomment-2506462937 I see, so in a nutshell we shouldn't be calling a fileIO close if it's still being referenced by any lingering tableOps ? maybe we can implement reference count, and only cal

Re: [PR] refine: refine interface of ManifestWriter [iceberg-rust]

ZENOTME commented on PR #738: URL: https://github.com/apache/iceberg-rust/pull/738#issuecomment-2506443194 cc @liurenjie1024 @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862406185 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean c

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862406185 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean c

Re: [I] Partition Spec Evolution API [iceberg-rust]

ZhiHanZ commented on issue #732: URL: https://github.com/apache/iceberg-rust/issues/732#issuecomment-2506361981 Can I take a trial on this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862410733 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862406185 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean c

Re: [PR] Materialized View Spec [iceberg]

JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1862370651 ## format/view-spec.md: ## @@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields: | _required_ | `representations` | A list of [re

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

Xuanwo closed issue #346: Tracking issue: Writing iceberg tables URL: https://github.com/apache/iceberg-rust/issues/346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

Xuanwo commented on issue #346: URL: https://github.com/apache/iceberg-rust/issues/346#issuecomment-2506325904 Tracked at https://github.com/apache/iceberg-rust/issues/700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Core: Propagate custom metrics reporter when table is created/replaced through Transaction [iceberg]

nastra merged PR #11671: URL: https://github.com/apache/iceberg/pull/11671 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Propagate custom metrics reporter when table is created/replaced through Transaction [iceberg]

nastra commented on PR #11671: URL: https://github.com/apache/iceberg/pull/11671#issuecomment-2506312843 thanks @jbonofre and @amogh-jahagirdar for the reviews -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

ZENOTME commented on issue #346: URL: https://github.com/apache/iceberg-rust/issues/346#issuecomment-2506308881 > @ZENOTME I missed this issue, sorry for that. Should we merge this one into #700 ? Sure! I think these are more about the writer. -- This is an automated message from t

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

Fokko commented on issue #346: URL: https://github.com/apache/iceberg-rust/issues/346#issuecomment-2506303131 @ZENOTME I missed this issue, sorry for that. Should we merge this one into #700 ? -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Logic to determine the partitions [iceberg-rust]

Fokko commented on issue #728: URL: https://github.com/apache/iceberg-rust/issues/728#issuecomment-2506301638 @liurenjie1024 I think this is what I meant with this issue: https://github.com/apache/iceberg-rust/issues/342 > but currently we are missing transaction api for adding data f

[I] Implement `initial-default` [iceberg-rust]

Fokko opened a new issue, #737: URL: https://github.com/apache/iceberg-rust/issues/737 Default values are part of the Iceberg specification V3: https://github.com/apache/iceberg/blob/main/format/spec.md#default-values But they are handy for reading the Iceberg metadata as well. If you

[I] Support for merge-commits [iceberg-rust]

Fokko opened a new issue, #736: URL: https://github.com/apache/iceberg-rust/issues/736 The so-called fast-appends are added in https://github.com/apache/iceberg-rust/pull/349 It would be good to also consider adding merge-commits. With the fast-append, a new manifest is written

Re: [I] Logic to determine the partitions [iceberg-rust]

liurenjie1024 commented on issue #728: URL: https://github.com/apache/iceberg-rust/issues/728#issuecomment-2506281161 I think @ZENOTME did some solid work for appending data files, but currently we are missing transaction api for adding data files. -- This is an automated message from the

Re: [PR] "how to release", add `gh` command to trigger workflow [iceberg-python]

ndrluis commented on PR #1387: URL: https://github.com/apache/iceberg-python/pull/1387#issuecomment-2506260295 @kevinjqliu Nice work! What do you think about having a Makefile with these commands? I think it would be easier than copying and pasting from the documentation. -- This is an a

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

jurossiar commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2506240731 Using this jar versions I can update iceberg tables: spark-version: 3.4 scala-version: 2.12 iceberg-version: 1.6.1 awssdk-version: 2.20.131 -- This is an automated mes

[I] Iceberg-rust Delete support [iceberg-rust]

Fokko opened a new issue, #735: URL: https://github.com/apache/iceberg-rust/issues/735 # Deletes For the deletes, we need a broader discussion on where the responsibilities lie between iceberg-rust and the query engine. On the read-side there [Tasks are passed to the query eng

Re: [I] Logic to determine the partitions [iceberg-rust]

Fokko closed issue #728: Logic to determine the partitions URL: https://github.com/apache/iceberg-rust/issues/728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] Logic to determine the partitions [iceberg-rust]

Fokko commented on issue #728: URL: https://github.com/apache/iceberg-rust/issues/728#issuecomment-2506215837 Let me close this one for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Core: Propagate custom metrics reporter when table is created/replaced through Transaction [iceberg]

amogh-jahagirdar commented on code in PR #11671: URL: https://github.com/apache/iceberg/pull/11671#discussion_r1862244838 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -368,6 +370,11 @@ public void renameView(TableIdentifier from, TableIdentifier

Re: [I] Able to parse name-mapping into a recusive structure. [iceberg-rust]

c-thiel commented on issue #723: URL: https://github.com/apache/iceberg-rust/issues/723#issuecomment-2506190962 @barronw gladly! Assigned the issue to you. If there are any questions, just post them here or contact us on Slack :) -- This is an automated message from the Apache Git Servi

Re: [I] Able to parse name-mapping into a recusive structure. [iceberg-rust]

barronw commented on issue #723: URL: https://github.com/apache/iceberg-rust/issues/723#issuecomment-2506162641 Can I pick this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Iceberg-rust Write support [iceberg-rust]

c-thiel commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2506154754 @barronw sure! Just comment in the one you prefer, then we can assign you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Iceberg-rust Write support [iceberg-rust]

barronw commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2506135632 @c-thiel @ZhiHanZ Can I pick up one of #723 or #732? #730 is already done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

Syed-bug commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2506095447 @jurossiar can you tell me the jars which u have used for 3.4.1 spark version for updating the iceberg table.In 3.4.1 version...I am getting this error...just wanna check on the ja

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1862157059 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) { f

Re: [PR] feat: support position delete writer [iceberg-rust]

ZENOTME commented on code in PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#discussion_r1862125870 ## crates/iceberg/src/writer/base_writer/position_delete_file_writer.rs: ## @@ -0,0 +1,320 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or m

Re: [PR] feat: support arrow_struct_to_iceberg_struct [iceberg-rust]

ZENOTME commented on PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#issuecomment-2506041190 cc @liurenjie1024 @Fokko @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

deniskuzZ commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2506025938 Found partition stats tracker issue https://github.com/apache/iceberg/issues/8450 with the following design doc: https://docs.google.com/document/d/1vaufuD47kMijz97LxM67X8OX-W2Wq7nmlz

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

deniskuzZ closed pull request #11669: Core: Support aggregated basic stats in partition summary URL: https://github.com/apache/iceberg/pull/11669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Bound Sort Order [iceberg-rust]

c-thiel closed issue #733: Bound Sort Order URL: https://github.com/apache/iceberg-rust/issues/733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issue

Re: [I] Implement the equality delete writer [iceberg-rust]

ZENOTME commented on issue #341: URL: https://github.com/apache/iceberg-rust/issues/341#issuecomment-2505946455 I this issue is completed after #703. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Iceberg-rust Write support [iceberg-rust]

c-thiel commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2505926549 @ZhiHanZ if you want an easy start, how about https://github.com/apache/iceberg-rust/issues/723? If you want something a bit more challenging, how about https://github.com/apac

Re: [I] Support to append file on table [iceberg-rust]

ZENOTME closed issue #345: Support to append file on table URL: https://github.com/apache/iceberg-rust/issues/345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Support to append file on table [iceberg-rust]

ZENOTME commented on issue #345: URL: https://github.com/apache/iceberg-rust/issues/345#issuecomment-2505944710 This issue is completed after https://github.com/apache/iceberg-rust/pull/349. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

ZENOTME commented on issue #346: URL: https://github.com/apache/iceberg-rust/issues/346#issuecomment-2505942835 > Thank you for compiling this @c-thiel @ZENOTME! I'm part of the Daft team and we're eagerly looking forward to write support. The tasks enumerated by @ZENOTME match our prioriti

[I] Sort Order Evolution API [iceberg-rust]

c-thiel opened a new issue, #733: URL: https://github.com/apache/iceberg-rust/issues/733 The general philosophy of any engine that implements Iceberg-Rust is that they should not care about things like field-IDs / source-ids. Therefore we try to hide this logic behind APIs. To allow evoluti

[I] Partition Spec Evolution API [iceberg-rust]

c-thiel opened a new issue, #732: URL: https://github.com/apache/iceberg-rust/issues/732 The general philosophy of any engine that implements Iceberg-Rust is that they should not care about things like field-IDs. Therefore we try to hide this logic behind APIs. To allow evolution of the par

[PR] feat: support arrow_struct_to_iceberg_struct [iceberg-rust]

ZENOTME opened a new pull request, #731: URL: https://github.com/apache/iceberg-rust/pull/731 This PR introduces the function to convert arrow struct to iceberg struct. This function is needed when we add fanout partition writer: In this writer, we need to compute the partition value using

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2505917957 @mxm Thank you very much for your suggestions. I need to add a version check in SortKeySerializerSnapshot. If the state is restored from an old version, I will directly return Typ

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

deniskuzZ commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2505896721 @pvary, unfortunately, that won't work. I was looking for an easy way to get basic partition stats, however, I missed the part that iceberg only keeps the changed partitions in a Snap

  1   2   >