Re: [PR] feat: transform function [iceberg-cpp]

2025-04-07 Thread via GitHub
lidavidm commented on code in PR #61: URL: https://github.com/apache/iceberg-cpp/pull/61#discussion_r2032469032 ## src/iceberg/transform/transform_spec.h: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agre

Re: [PR] feat: add an AddFieldsAction to transaction [iceberg-rust]

2025-04-07 Thread via GitHub
cmcarthur commented on code in PR #1176: URL: https://github.com/apache/iceberg-rust/pull/1176#discussion_r2031711629 ## crates/iceberg/src/transaction/add_field.rs: ## @@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor licen

Re: [PR] fix: TableMetadata max sequence number validation [iceberg-rust]

2025-04-07 Thread via GitHub
c-thiel commented on code in PR #1167: URL: https://github.com/apache/iceberg-rust/pull/1167#discussion_r2031790551 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -558,6 +558,24 @@ impl TableMetadata { )); } +if self.format_version >= FormatV

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
divinerapier closed pull request #1153: feat(io): add OSS storage implementation and update dependencies URL: https://github.com/apache/iceberg-rust/pull/1153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
divinerapier commented on PR #1153: URL: https://github.com/apache/iceberg-rust/pull/1153#issuecomment-2784995101 > There are no good OSS-compatible services like MinIO for S3 that suit our testing needs. OpenDAL tests OSS using a user-sponsored, dedicated testing bucket. Perhaps we should

[PR] chore(deps): Bump tokio from 1.44.1 to 1.44.2 [iceberg-rust]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1179: URL: https://github.com/apache/iceberg-rust/pull/1179 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.44.1 to 1.44.2. Release notes Sourced from https://github.com/tokio-rs/tokio/releases";>tokio's releases. Tokio v1.4

Re: [PR] feat: Infer partition values from bounds [iceberg-rust]

2025-04-07 Thread via GitHub
jonathanc-n commented on PR #1079: URL: https://github.com/apache/iceberg-rust/pull/1079#issuecomment-2784874054 @liurenjie1024 Good catch, should be good now. Thank you for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
divinerapier commented on code in PR #1153: URL: https://github.com/apache/iceberg-rust/pull/1153#discussion_r2032312062 ## crates/iceberg/src/io/storage.rs: ## @@ -45,6 +47,17 @@ pub(crate) enum Storage { client: reqwest::Client, config: Arc, }, +#[cf

Re: [PR] Add datafusion cli for iceberg [iceberg-rust]

2025-04-07 Thread via GitHub
xxchan commented on code in PR #1143: URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2032308752 ## rust-toolchain.toml: ## Review Comment: I don't have strong opinions on this. It would be nice to have more frequent iceberg-rs releases. But it se

Re: [PR] Change tokio feature [iceberg-rust]

2025-04-07 Thread via GitHub
Xuanwo commented on PR #1173: URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785060418 > WDYT if I change full to `rt-multithread` Makes more sense to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Change tokio feature [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on PR #1173: URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785051178 > I suggest we wait a bit to see how #1036 progresses. Ideally, all dependencies on `cfg(tokio)` can be eliminated once our mini engine has been extracted. @Xuanwo Seem

[PR] chore(deps): Bump tokio from 1.43.0 to 1.44.2 in /bindings/python [iceberg-rust]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1180: URL: https://github.com/apache/iceberg-rust/pull/1180 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.43.0 to 1.44.2. Release notes Sourced from https://github.com/tokio-rs/tokio/releases";>tokio's releases. Tokio v1.4

Re: [I] Why isn't a lock added when writing Version Hint in HadoopTableOperations [iceberg]

2025-04-07 Thread via GitHub
404921490 commented on issue #12740: URL: https://github.com/apache/iceberg/issues/12740#issuecomment-2785043692 Thank you again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Why isn't a lock added when writing Version Hint in HadoopTableOperations [iceberg]

2025-04-07 Thread via GitHub
404921490 closed issue #12740: Why isn't a lock added when writing Version Hint in HadoopTableOperations URL: https://github.com/apache/iceberg/issues/12740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Change tokio feature [iceberg-rust]

2025-04-07 Thread via GitHub
Xuanwo commented on PR #1173: URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785035590 I suggest we wait a bit to see how https://github.com/apache/iceberg-rust/issues/1036 progresses. Ideally, all dependencies on `cfg(tokio)` can be eliminated once our mini engine has

Re: [PR] feat: snapshot [iceberg-cpp]

2025-04-07 Thread via GitHub
lidavidm commented on code in PR #60: URL: https://github.com/apache/iceberg-cpp/pull/60#discussion_r2032252983 ## src/iceberg/snapshot.h: ## @@ -0,0 +1,217 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the

Re: [PR] feat(puffin): Make Puffin APIs public [iceberg-rust]

2025-04-07 Thread via GitHub
fqaiser94 commented on code in PR #1165: URL: https://github.com/apache/iceberg-rust/pull/1165#discussion_r2032210965 ## crates/iceberg/src/puffin/metadata.rs: ## @@ -26,33 +26,33 @@ use crate::{Error, ErrorKind, Result}; /// Human-readable identification of the application w

Re: [PR] Alternative implementation for building and testing hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-07 Thread via GitHub
wypoon commented on code in PR #12721: URL: https://github.com/apache/iceberg/pull/12721#discussion_r2032172207 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java: ## @@ -186,9 +189,9 @@ public void stop() throws Exception { if (executorService !

[PR] Build: Bump typing-extensions from 4.12.2 to 4.13.1 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1897: URL: https://github.com/apache/iceberg-python/pull/1897 Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.12.2 to 4.13.1. Release notes Sourced from https://github.com/python/typing_extensions/releases";>

[I] RUSTSEC-2025-0023: Broadcast channel calls clone in parallel, but does not require `Sync` [iceberg-rust]

2025-04-07 Thread via GitHub
github-actions[bot] opened a new issue, #1178: URL: https://github.com/apache/iceberg-rust/issues/1178 > Broadcast channel calls clone in parallel, but does not require `Sync` | Details || | --- | --

[PR] Build: Bump tenacity from 9.0.0 to 9.1.2 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1896: URL: https://github.com/apache/iceberg-python/pull/1896 Bumps [tenacity](https://github.com/jd/tenacity) from 9.0.0 to 9.1.2. Release notes Sourced from https://github.com/jd/tenacity/releases";>tenacity's releases. 9.1.2 F

[PR] Build: Bump mkdocs-section-index from 0.3.9 to 0.3.10 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1895: URL: https://github.com/apache/iceberg-python/pull/1895 Bumps [mkdocs-section-index](https://github.com/oprypin/mkdocs-section-index) from 0.3.9 to 0.3.10. Release notes Sourced from https://github.com/oprypin/mkdocs-section-index/

Re: [I] Docs: REST OAuth2 Client Authentication Guide [iceberg]

2025-04-07 Thread via GitHub
github-actions[bot] commented on issue #11286: URL: https://github.com/apache/iceberg/issues/11286#issuecomment-2784909872 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2025-04-07 Thread via GitHub
github-actions[bot] closed pull request #11369: Add scan planning api request and response models, parsers URL: https://github.com/apache/iceberg/pull/11369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump griffe from 1.7.1 to 1.7.2 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1894: URL: https://github.com/apache/iceberg-python/pull/1894 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 1.7.1 to 1.7.2. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

Re: [PR] Alternative implementation for building and testing hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-07 Thread via GitHub
wypoon commented on code in PR #12721: URL: https://github.com/apache/iceberg/pull/12721#discussion_r2032166743 ## hive3-metastore/src/test/java/org/apache/iceberg/hive/TestHiveClientPool.java: ## @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

[PR] Build: Bump mkdocstrings-python from 1.16.8 to 1.16.10 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1893: URL: https://github.com/apache/iceberg-python/pull/1893 Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.16.8 to 1.16.10. Release notes Sourced from https://github.com/mkdocstrings/python/releases";>mkdocst

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-07 Thread via GitHub
wypoon commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2032163482 ## hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java: ## @@ -132,7 +131,7 @@ public R run(Action action, boolean retry) static Key extrac

[PR] Build: Bump mkdocs-material from 9.6.10 to 9.6.11 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1891: URL: https://github.com/apache/iceberg-python/pull/1891 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.10 to 9.6.11. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

[PR] Build: Bump moto from 5.1.1 to 5.1.3 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1889: URL: https://github.com/apache/iceberg-python/pull/1889 Bumps [moto](https://github.com/getmoto/moto) from 5.1.1 to 5.1.3. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.

[PR] Build: Bump getdaft from 0.4.8 to 0.4.9 [iceberg-python]

2025-04-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1890: URL: https://github.com/apache/iceberg-python/pull/1890 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.4.8 to 0.4.9. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-07 Thread via GitHub
wypoon commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2032146553 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java: ## @@ -303,10 +303,10 @@ public void testThriftExceptionConcurrentCommit() throws TExcep

[PR] feat: Support tag and branch operations [iceberg-rust]

2025-04-07 Thread via GitHub
jonathanc-n opened a new pull request, #1177: URL: https://github.com/apache/iceberg-rust/pull/1177 ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Added `ManageSnapshots` for basic operations. This references the python implem

Re: [I] "manylinux_2_34_aarch64" wheel request [iceberg-python]

2025-04-07 Thread via GitHub
Fokko commented on issue #1807: URL: https://github.com/apache/iceberg-python/issues/1807#issuecomment-2784598371 I was also playing around with this last week, but it it turns out that it will blow up the tests up to 2h+: https://github.com/Fokko/iceberg-python/actions/runs/14219816306/jo

[I] Pip install with glue and s3fs extras is not possible from 0.9.0 [iceberg-python]

2025-04-07 Thread via GitHub
cshenrik opened a new issue, #1888: URL: https://github.com/apache/iceberg-python/issues/1888 ### Question I'm using Poetry to build a library that uses Pyiceberg. I have `pyiceberg[glue,s3fs]` as a dependency. The users of my library install it using pip. This has all worked fine un

Re: [PR] Flink: Dynamic Iceberg Sink Contribution [iceberg]

2025-04-07 Thread via GitHub
mxm commented on PR #12424: URL: https://github.com/apache/iceberg/pull/12424#issuecomment-2784587046 Quick recap on a discussion Steven, Peter, and I had in a separate channel, which is only partially captured in the PR comments: ### Iceberg Schema Dynamic Sink users currently

[PR] Upgrades Flink Version to 1.19.2 and 1.20.1 [iceberg]

2025-04-07 Thread via GitHub
rodmeneses opened a new pull request, #12745: URL: https://github.com/apache/iceberg/pull/12745 This code also updates `testRecoveryFromSnapshotWithoutCompletedNotification` unit test, by removing the second assertion when we check that `harness.initializeState` is actually committing pend

Re: [PR] fix: TableMetadata max sequence number validation [iceberg-rust]

2025-04-07 Thread via GitHub
c-thiel commented on code in PR #1167: URL: https://github.com/apache/iceberg-rust/pull/1167#discussion_r2031790551 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -558,6 +558,24 @@ impl TableMetadata { )); } +if self.format_version >= FormatV

Re: [PR] allow string transforms in add_field() [iceberg-python]

2025-04-07 Thread via GitHub
Federico-Vichachi commented on PR #1883: URL: https://github.com/apache/iceberg-python/pull/1883#issuecomment-2784186020 Hi @Fokko, I appreciate your corrections and will include them. Regarding the test, I was trying to run the following one but had problems installing the project's depend

Re: [PR] Core: Enable row lineage for all v3 tables [iceberg]

2025-04-07 Thread via GitHub
rdblue commented on PR #12593: URL: https://github.com/apache/iceberg/pull/12593#issuecomment-2783997013 Now that the vote has passed, I'll merge this. I'm following up in #12672 to add more core functionality so I'll fix the `TableMetadata` return value in that PR. Thanks for the reviews,

Re: [PR] feat: add an AddFieldsAction to transaction [iceberg-rust]

2025-04-07 Thread via GitHub
cmcarthur closed pull request #1175: feat: add an AddFieldsAction to transaction URL: https://github.com/apache/iceberg-rust/pull/1175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] feat (catalog/rest): Add create view for rest catalog [iceberg-go]

2025-04-07 Thread via GitHub
dttung2905 commented on PR #376: URL: https://github.com/apache/iceberg-go/pull/376#issuecomment-2784047531 > This looks good. As a separate PR can we look into adding a test to the rest_integration_test.go file to ensure we have integration tests with the rest catalog? That makes to

Re: [I] Implement expire snapshots maintenance operation [iceberg-go]

2025-04-07 Thread via GitHub
arnaudbriche commented on issue #367: URL: https://github.com/apache/iceberg-go/issues/367#issuecomment-2784015833 Hi @zeroshade ! I'm working on implementing snapshot expiration. Looking at how `MetadataBuilder` and `Update` works, I'm a bit confused about how things are suppo

Re: [PR] Core: Enable row lineage for all v3 tables [iceberg]

2025-04-07 Thread via GitHub
rdblue merged PR #12593: URL: https://github.com/apache/iceberg/pull/12593 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] AWS: Add AWS crt client support [iceberg]

2025-04-07 Thread via GitHub
sullis commented on PR #10217: URL: https://github.com/apache/iceberg/pull/10217#issuecomment-2783986291 for posterity: Apache Iceberg Java main branch has support for AWS S3 CRT https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/DefaultS

Re: [PR] fix(table): for add partition spec constant [iceberg-go]

2025-04-07 Thread via GitHub
mimikwang commented on code in PR #382: URL: https://github.com/apache/iceberg-go/pull/382#discussion_r2031616927 ## table/metadata.go: ## @@ -38,7 +38,7 @@ const ( partitionFieldStartID = 1000 supportedTableFormatVersion = 2 - addPartionSpecAction

Re: [PR] feat (catalog/rest): Add create view for rest catalog [iceberg-go]

2025-04-07 Thread via GitHub
dttung2905 commented on code in PR #376: URL: https://github.com/apache/iceberg-go/pull/376#discussion_r2031604779 ## catalog/rest/rest.go: ## @@ -1156,3 +1156,80 @@ func (r *Catalog) CheckViewExists(ctx context.Context, identifier table.Identifi return true, nil } +

Re: [PR] feat (catalog/rest): Add create view for rest catalog [iceberg-go]

2025-04-07 Thread via GitHub
dttung2905 commented on PR #376: URL: https://github.com/apache/iceberg-go/pull/376#issuecomment-2783937743 > I'd love it if you pursued implementing the View objects. Please create `view` as a separate package from table though. Sure. I will try to find some time this week to impleme

Re: [PR] AWS: Fix DynamoDB and Glue integration test failures [iceberg]

2025-04-07 Thread via GitHub
danielcweeks commented on code in PR #12718: URL: https://github.com/apache/iceberg/pull/12718#discussion_r2031518904 ## aws/src/integration/java/org/apache/iceberg/aws/dynamodb/TestDynamoDbLockManager.java: ## @@ -128,7 +128,10 @@ public void testAcquireOnceMultiProcesses() thr

Re: [I] Merge operation in spark could not find column [iceberg]

2025-04-07 Thread via GitHub
rami-lv commented on issue #12661: URL: https://github.com/apache/iceberg/issues/12661#issuecomment-2783870046 I will give it a try thanks for getting back on this. I wonder why the exception message doesn't mention that. -- This is an automated message from the Apache Git Service. To re

Re: [I] [EPIC] Iceberg-rust Write support [iceberg-rust]

2025-04-07 Thread via GitHub
kevinjqliu commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2783868421 @ion-elgreco great to see you here! big fan of delta-rs and its python bindings :) This issue just tracks write support for the rust project. > is there a clear

Re: [PR] AWS: Fix DynamoDB and Glue integration test failures [iceberg]

2025-04-07 Thread via GitHub
lliangyu-lin commented on code in PR #12718: URL: https://github.com/apache/iceberg/pull/12718#discussion_r2031550204 ## aws/src/integration/java/org/apache/iceberg/aws/dynamodb/TestDynamoDbLockManager.java: ## @@ -128,7 +128,10 @@ public void testAcquireOnceMultiProcesses() thr

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-07 Thread via GitHub
pvary commented on PR #12629: URL: https://github.com/apache/iceberg/pull/12629#issuecomment-2783731812 > > Do we expect someone just using the calculation and someone using the stat file generation too? Or it was just a technical limitation? > > @pvary, I think the original design tr

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-07 Thread via GitHub
pvary commented on PR #12692: URL: https://github.com/apache/iceberg/pull/12692#issuecomment-2783741576 @danielcweeks, @manuzhang, @RussellSpitzer: I think this PR is ready for review now. Sorry for the force push. Instead of creating new files, I renamed the old ones so it is easier to rev

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-07 Thread via GitHub
pvary commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2031502283 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -1720,143 +1717,15 @@ public void testInvalidAPIUsage() {

Re: [PR] Added support for evolving the partition of the table [iceberg]

2025-04-07 Thread via GitHub
danielcweeks commented on PR #12723: URL: https://github.com/apache/iceberg/pull/12723#issuecomment-2783728529 @kumarpritam863 Can you help me understand what the use case is here? I feel like this approach is prone to causing issues when managing multiple tables. Why not just evolve the

[PR] feat: snapshot [iceberg-cpp]

2025-04-07 Thread via GitHub
zhjwpku opened a new pull request, #60: URL: https://github.com/apache/iceberg-cpp/pull/60 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [I] External CatalogIO implementation [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade closed issue #377: External CatalogIO implementation URL: https://github.com/apache/iceberg-go/issues/377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] feat(catalog): Make CreateTableOpt public to ease creation of custom Catalog Impls [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade merged PR #380: URL: https://github.com/apache/iceberg-go/pull/380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat(catalog): Make CreateTableOpt public to ease creation of custom Catalog Impls [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade commented on code in PR #380: URL: https://github.com/apache/iceberg-go/pull/380#discussion_r2031463740 ## catalog/catalog.go: ## @@ -67,6 +67,7 @@ type PropertiesUpdateSummary struct { Missing []string `json:"missing"` } +// Optional config of CreateTable o

Re: [PR] Make CreateTableOpt public to ease creation of custom CatalogIO imple… [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade commented on code in PR #380: URL: https://github.com/apache/iceberg-go/pull/380#discussion_r2031443143 ## catalog/catalog.go: ## @@ -68,6 +67,13 @@ type PropertiesUpdateSummary struct { Missing []string `json:"missing"` } +type CreateTableCfg struct { Revi

Re: [PR] fix(table): for add partition spec constant [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade commented on code in PR #382: URL: https://github.com/apache/iceberg-go/pull/382#discussion_r2031452329 ## table/metadata.go: ## @@ -38,7 +38,7 @@ const ( partitionFieldStartID = 1000 supportedTableFormatVersion = 2 - addPartionSpecAction

Re: [PR] feat (catalog/rest): Add create view for rest catalog [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade commented on code in PR #376: URL: https://github.com/apache/iceberg-go/pull/376#discussion_r2031448057 ## catalog/rest/rest.go: ## @@ -1156,3 +1156,80 @@ func (r *Catalog) CheckViewExists(ctx context.Context, identifier table.Identifi return true, nil } +

Re: [PR] feat (catalog/rest): Add create view for rest catalog [iceberg-go]

2025-04-07 Thread via GitHub
zeroshade commented on PR #376: URL: https://github.com/apache/iceberg-go/pull/376#issuecomment-2783647259 I'd love it if you pursued implementing the View objects. Please create `view` as a separate package from table though. -- This is an automated message from the Apache Git Service.

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-07 Thread via GitHub
deniskuzZ commented on PR #12629: URL: https://github.com/apache/iceberg/pull/12629#issuecomment-2783367679 > Do we expect someone just using the calculation and someone using the stat file generation too? Or it was just a technical limitation? I think the original design tried to separat

Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]

2025-04-07 Thread via GitHub
arnaudbriche commented on issue #763: URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2783472021 I submitted a small PR to allow using `version-hint.text`: https://github.com/apache/iceberg-python/pull/1887 -- This is an automated message from the Apache Git Service.

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-07 Thread via GitHub
lidavidm commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2031193620 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See t

Re: [PR] Change tokio feature [iceberg-rust]

2025-04-07 Thread via GitHub
Xuanwo commented on PR #1173: URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2783462518 `tokio/full` is really big. Do we really want it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-07 Thread via GitHub
yingcai-cy commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2031159579 ## src/iceberg/error.h: ## @@ -38,6 +38,8 @@ enum class ErrorKind { kNotImplemented, kUnknownError, kNotSupported, + kInvalidExpression, + kInvalidOperato

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-07 Thread via GitHub
pvary commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2031216069 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -149,6 +162,92 @@ public static PartitionStatisticsFile computeAndWriteStatsFile(Table

Re: [I] Arrow uint32/64 support [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on issue #1164: URL: https://github.com/apache/iceberg-rust/issues/1164#issuecomment-2783302176 Thanks @hugokitano for raising this. Iceberg has no support for non negative data types, I think user can convert them to corresponding signed data types before writing?

Re: [PR] feat: Infer partition values from bounds [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1079: URL: https://github.com/apache/iceberg-rust/pull/1079#discussion_r2031258473 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -458,6 +459,54 @@ impl ParquetWriter { Ok(builder) } + +#[allow(dead_co

Re: [PR] Scan Delete Support Part 3: `ArrowReader::build_deletes_row_selection` implementation [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 merged PR #951: URL: https://github.com/apache/iceberg-rust/pull/951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: re-export name mapping [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1116: URL: https://github.com/apache/iceberg-rust/pull/1116#discussion_r2031239672 ## crates/iceberg/src/spec/name_mapping.rs: ## @@ -33,12 +48,38 @@ pub struct NameMapping { #[serde(rename_all = "kebab-case")] pub struct MappedField {

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-07 Thread via GitHub
pvary commented on PR #12629: URL: https://github.com/apache/iceberg/pull/12629#issuecomment-2783310921 > @pvary, i don't follow where you are proposing these API changes? > [..] > If you are proposing to implement these API in `PartitionStatsUtil`, it would be problematic since it doe

Re: [PR] Flink:Backport fix npe in SketchUtil to Flink 1.19 and 1.18 [iceberg]

2025-04-07 Thread via GitHub
Guosmilesmile commented on PR #12741: URL: https://github.com/apache/iceberg/pull/12741#issuecomment-2783329296 @pvary Peter, Thank you for your reviews -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Change tokio feature [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 opened a new pull request, #1173: URL: https://github.com/apache/iceberg-rust/pull/1173 ## What changes are included in this PR? Changes tokio dependency, enable `sync` feature only by default, but requires `full` feature when we enable `tokio` cfg. With this change, we

Re: [PR] Flink:Backport fix npe in SketchUtil to Flink 1.19 and 1.18 [iceberg]

2025-04-07 Thread via GitHub
pvary merged PR #12741: URL: https://github.com/apache/iceberg/pull/12741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink:Backport fix npe in SketchUtil to Flink 1.19 and 1.18 [iceberg]

2025-04-07 Thread via GitHub
pvary commented on PR #12741: URL: https://github.com/apache/iceberg/pull/12741#issuecomment-2783323514 Merged to main. Thanks for the backport @Guosmilesmile! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Why isn't a lock added when writing Version Hint in HadoopTableOperations [iceberg]

2025-04-07 Thread via GitHub
pvary commented on issue #12740: URL: https://github.com/apache/iceberg/issues/12740#issuecomment-2783318473 > [..] Can we assume that once the snapshot file is successfully created, it means the commit is successful Yes. Be aware that the community considers deprecating the Ha

Re: [PR] Flink:Backport fix npe in SketchUtil to Flink 1.19 and 1.18 [iceberg]

2025-04-07 Thread via GitHub
Guosmilesmile commented on PR #12741: URL: https://github.com/apache/iceberg/pull/12741#issuecomment-2783318171 @pvary This is a clearn backport. Please take some time to review it , Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] feat(puffin): Make Puffin APIs public [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1165: URL: https://github.com/apache/iceberg-rust/pull/1165#discussion_r2031166705 ## crates/iceberg/src/puffin/metadata.rs: ## @@ -91,13 +91,13 @@ impl Flag { /// Metadata about a puffin file. /// For more information, see: https://iceb

Re: [PR] fix: TableMetadata max sequence number validation [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1167: URL: https://github.com/apache/iceberg-rust/pull/1167#discussion_r2031213325 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -558,6 +558,24 @@ impl TableMetadata { )); } +if self.format_version >= F

Re: [I] Make header X-Iceberg-Access-Delegation optional conforming with the Iceberg REST api spec [iceberg-python]

2025-04-07 Thread via GitHub
alzio2607 commented on issue #1886: URL: https://github.com/apache/iceberg-python/issues/1886#issuecomment-2783231881 Turns out, you **_can_** skip the header by specifying it as None in your load catalog configurations like so: ``` catalog = load_catalog( 'default',

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
Xuanwo commented on PR #1153: URL: https://github.com/apache/iceberg-rust/pull/1153#issuecomment-2782679433 > I am in the process of implementing support for Aliyun OSS storage backend using OpenDAL, can you provide me with some advice on how to test this? There are no good OSS-compat

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-07 Thread via GitHub
lidavidm commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2031195685 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See t

Re: [PR] Spark 3.5 row lineage [iceberg]

2025-04-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2031189720 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,266 @@ +/* + * L

Re: [PR] Spark 3.5 row lineage [iceberg]

2025-04-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2031189720 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,266 @@ +/* + * L

Re: [I] Make header X-Iceberg-Access-Delegation optional conforming with the Iceberg REST api spec [iceberg-python]

2025-04-07 Thread via GitHub
alzio2607 closed issue #1886: Make header X-Iceberg-Access-Delegation optional conforming with the Iceberg REST api spec URL: https://github.com/apache/iceberg-python/issues/1886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [I] Why isn't a lock added when writing Version Hint in HadoopTableOperations [iceberg]

2025-04-07 Thread via GitHub
404921490 commented on issue #12740: URL: https://github.com/apache/iceberg/issues/12740#issuecomment-2783211563 ok,i see,Thanks for your reply。Can we assume that once the snapshot file is successfully created, it means the commit is successful -- This is an automated message from the Apa

Re: [I] Glue Catalog updating [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 closed issue #1169: Glue Catalog updating URL: https://github.com/apache/iceberg-rust/issues/1169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Glue Catalog updating [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on issue #1169: URL: https://github.com/apache/iceberg-rust/issues/1169#issuecomment-2783212685 I'll close this as it duplicates with #1168 , feel free to reopen it if necessary. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [D] Glue catalog updating [iceberg-rust]

2025-04-07 Thread via GitHub
GitHub user liurenjie1024 added a comment to the discussion: Glue catalog updating Hi, @hugokitano updating table is under development, it's not fully delivered. GitHub link: https://github.com/apache/iceberg-rust/discussions/1168#discussioncomment-12750540 This is an automatically sent

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-07 Thread via GitHub
yingcai-cy commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2031157918 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] Add datafusion cli for iceberg [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1143: URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2031156783 ## rust-toolchain.toml: ## Review Comment: > (imagine datafusion releases more frequently, and we can't bump MSRV until next release.) Do we ne

Re: [PR] chore(deps): Bump crate-ci/typos from 1.31.0 to 1.31.1 [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 merged PR #1171: URL: https://github.com/apache/iceberg-rust/pull/1171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1153: URL: https://github.com/apache/iceberg-rust/pull/1153#discussion_r2031144720 ## crates/iceberg/src/io/storage_oss.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-07 Thread via GitHub
yingcai-cy commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2031143729 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-07 Thread via GitHub
liurenjie1024 commented on code in PR #1153: URL: https://github.com/apache/iceberg-rust/pull/1153#discussion_r2031143238 ## crates/iceberg/src/io/storage.rs: ## @@ -45,6 +47,17 @@ pub(crate) enum Storage { client: reqwest::Client, config: Arc, }, +#[c

[I] Make header X-Iceberg-Access-Delegation optional conforming with the Iceberg REST api spec [iceberg-python]

2025-04-07 Thread via GitHub
alzio2607 opened a new issue, #1886: URL: https://github.com/apache/iceberg-python/issues/1886 ### Feature Request / Improvement According to Iceberg's [Rest catalog spec](https://github.com/apache/iceberg/blob/b428fbc59bd1579f4dc918a5cd48fce667d81ce1/open-api/rest-catalog-open-api.ya

  1   2   >