Re: [PR] Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing [iceberg-rust]

2025-05-22 Thread via GitHub
sdd commented on PR #982: URL: https://github.com/apache/iceberg-rust/pull/982#issuecomment-2903386196 Hi @liurenjie1024 / @Xuanwo / @xxchan. This is now ready again for review after a refactor taking into account @xxchan's great feedback. I'll be on holiday for a week after today so

Re: [PR] feat: support decompress gzip metadata [iceberg-cpp]

2025-05-22 Thread via GitHub
yingcai-cy commented on code in PR #108: URL: https://github.com/apache/iceberg-cpp/pull/108#discussion_r2103886619 ## src/iceberg/table_metadata.cc: ## @@ -153,14 +154,70 @@ Result TableMetadataUtil::CodecFromFileName( return MetadataFileCodecType::kNone; } +class GZipDe

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-22 Thread via GitHub
emkornfield commented on PR #12598: URL: https://github.com/apache/iceberg/pull/12598#issuecomment-2903372715 > @emkornfield Did we ever finish this vote? Sorry for the delay, just closed the vote as passing -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Encryption integration and test [iceberg]

2025-05-22 Thread via GitHub
ggershinsky commented on code in PR #13066: URL: https://github.com/apache/iceberg/pull/13066#discussion_r2103835988 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -124,13 +184,37 @@ protected void doRefresh() { throw new RuntimeEx

[PR] Error handling with DLQ support [iceberg]

2025-05-22 Thread via GitHub
devendra-nr opened a new pull request, #13135: URL: https://github.com/apache/iceberg/pull/13135 Error handling with DLQ support -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Error handling with DLQ support [iceberg]

2025-05-22 Thread via GitHub
devendra-nr closed pull request #13135: Error handling with DLQ support URL: https://github.com/apache/iceberg/pull/13135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2103809789 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/BaseLockFactoryBuilder.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Soft

Re: [PR] feat(storage-azdls): Add Azure Datalake Storage support [iceberg-rust]

2025-05-22 Thread via GitHub
Xuanwo commented on PR #1368: URL: https://github.com/apache/iceberg-rust/pull/1368#issuecomment-2903270335 Nice work, @DerGut! Most of the changes in this PR look good to me. We can merge it after the next opendal release, which will include most of what we need here. -- This is an auto

Re: [I] Tracking issues of Iceberg Rust 0.5.0 Release (May 2025) [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #1325: URL: https://github.com/apache/iceberg-rust/issues/1325#issuecomment-2903266950 0.5.0 RC2 devlist thread, https://lists.apache.org/thread/8q5jtyd6ykztj5wx514wx3wj9hnwlbpf -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] Tracking issues of Iceberg Rust 0.5.0 Release (May 2025) [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #1325: URL: https://github.com/apache/iceberg-rust/issues/1325#issuecomment-2903254726 Create ASF release ``` git checkout v0.5.0-rc.2 ICEBERG_VERSION=0.5.0 ICEBERG_VERSION_RC=2 ./scripts/release.sh svn co https://dist.apache.org/repos/dist/dev/ic

Re: [I] Tracking issues of Iceberg Rust 0.5.0 Release (May 2025) [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #1325: URL: https://github.com/apache/iceberg-rust/issues/1325#issuecomment-2903248428 #1371 is merged. Using [`05969b3` commit](https://github.com/apache/iceberg-rust/commit/05969b38451ef13026b71f75693efc72219e7cad) for `v0.5.0-rc.2` ``` git tag

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2103766896 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/BaseLockFactoryBuilder.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Soft

Re: [PR] feat: add asan and ubsan support to cmake [iceberg-cpp]

2025-05-22 Thread via GitHub
lidavidm commented on code in PR #107: URL: https://github.com/apache/iceberg-cpp/pull/107#discussion_r2103733217 ## .github/workflows/sanitizer_test.yml: ## @@ -0,0 +1,57 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] feat: add asan and ubsan support to cmake [iceberg-cpp]

2025-05-22 Thread via GitHub
wgtmac commented on PR #107: URL: https://github.com/apache/iceberg-cpp/pull/107#issuecomment-2903140398 ``` [ 0%] Building C object _deps/nanoarrow-build/CMakeFiles/nanoarrow.dir/src/nanoarrow/common/array.c.o cd /home/runner/work/iceberg-cpp/iceberg-cpp/build/_deps/nanoarrow-build &

[PR] feat: support decompress gzip metadata [iceberg-cpp]

2025-05-22 Thread via GitHub
dongxiao1198 opened a new pull request, #108: URL: https://github.com/apache/iceberg-cpp/pull/108 support gzip compressed metadata json -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] [Feat] Add rewrite_table_path Spark procedure to support table replication [iceberg-python]

2025-05-22 Thread via GitHub
sungwy commented on issue #2014: URL: https://github.com/apache/iceberg-python/issues/2014#issuecomment-2903131196 Thanks for raising this feature request @abfisher0417 ! I agree with @corleyma I think operations that manipulate Iceberg metadata are good feature candidates for PyIc

Re: [PR] feat: add asan and ubsan support to cmake [iceberg-cpp]

2025-05-22 Thread via GitHub
wgtmac commented on PR #107: URL: https://github.com/apache/iceberg-cpp/pull/107#issuecomment-2903123846 @lidavidm @raulcd Could you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Make FileIO a Trait [iceberg-rust]

2025-05-22 Thread via GitHub
linhr commented on issue #1314: URL: https://github.com/apache/iceberg-rust/issues/1314#issuecomment-2903122746 > That's the problem, we are defining a richer interface that's not used by iceberg, then it would be confusing for people who want to define their own FileIO implementation, whic

Re: [PR] Add Avro compression [iceberg-python]

2025-05-22 Thread via GitHub
sungwy commented on code in PR #1976: URL: https://github.com/apache/iceberg-python/pull/1976#discussion_r2103706160 ## pyiceberg/table/update/snapshot.py: ## @@ -126,6 +128,11 @@ def __init__( self._deleted_data_files = set() self.snapshot_properties = snapsho

Re: [I] Issue when connecting to REST catalogs on AWS ( Amazon SageMaker Lakehouse) [iceberg-python]

2025-05-22 Thread via GitHub
davidshtian commented on issue #1449: URL: https://github.com/apache/iceberg-python/issues/1449#issuecomment-2903101031 @lazara2 Thanks for your help~ The docs here are really helpful! I've tried using Glue Extension endpoint, and yes the `list_tables` showed nothing. Then I found th

Re: [I] Cannot create a table from a pyarrow schema [iceberg-python]

2025-05-22 Thread via GitHub
sungwy commented on issue #2030: URL: https://github.com/apache/iceberg-python/issues/2030#issuecomment-2903096164 Hi @DavidEscott thanks for raising this issue! As you pointed out this has been a frequently discussed issue on PyIceberg, and there's still the original open issue that track

Re: [I] Iceberg Table and View Unification Proposal [iceberg]

2025-05-22 Thread via GitHub
monologuist commented on issue #13126: URL: https://github.com/apache/iceberg/issues/13126#issuecomment-2903088308 Yes, I am a real human. Sorry, I did use AI to assist me in expressing myself in the proposal. But this is a problem in our real production scenario. The current interface

Re: [PR] Change `FieldSummary` `{upper,lower}_bound` to `ByteBuf` [iceberg-rust]

2025-05-22 Thread via GitHub
Xuanwo commented on code in PR #1369: URL: https://github.com/apache/iceberg-rust/pull/1369#discussion_r2103677477 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -154,9 +155,19 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { _predicate: &

Re: [PR] fix: add support for `Decimal` and `Uuid` datum conversion [iceberg-rust]

2025-05-22 Thread via GitHub
Xuanwo merged PR #1346: URL: https://github.com/apache/iceberg-rust/pull/1346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.37.0 to 2.38.0 [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on PR #12852: URL: https://github.com/apache/iceberg/pull/12852#issuecomment-2903059389 not stale -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] add new commits to changelog for 0.5.0 [iceberg-rust]

2025-05-22 Thread via GitHub
Xuanwo merged PR #1371: URL: https://github.com/apache/iceberg-rust/pull/1371 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Implement schema update [iceberg-go]

2025-05-22 Thread via GitHub
zeroshade commented on issue #430: URL: https://github.com/apache/iceberg-go/issues/430#issuecomment-2902958475 I've been super busy but I promise I'll give this a review tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] fix: add metadata_properties to _construct_parameters when update hive table [iceberg-python]

2025-05-22 Thread via GitHub
geruh commented on code in PR #2013: URL: https://github.com/apache/iceberg-python/pull/2013#discussion_r2103610695 ## pyiceberg/catalog/hive.py: ## @@ -211,11 +211,18 @@ def _construct_hive_storage_descriptor( DEFAULT_PROPERTIES = {TableProperties.PARQUET_COMPRESSION: TablePr

Re: [PR] AWS: pass storage credentials to s3 client [iceberg]

2025-05-22 Thread via GitHub
plusplusjiajia commented on PR #13118: URL: https://github.com/apache/iceberg/pull/13118#issuecomment-2902953148 > #12799 has been merged before I had the chance to merge your PR. Can you rebase this please so that we can get the tests in sure, I've reverted the change. -- This is

Re: [PR] Core, Data: File Format API interfaces [iceberg]

2025-05-22 Thread via GitHub
stevenzwu commented on code in PR #12774: URL: https://github.com/apache/iceberg/pull/12774#discussion_r2103603849 ## core/src/main/java/org/apache/iceberg/io/WriteBuilder.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [I] Implement schema update [iceberg-go]

2025-05-22 Thread via GitHub
xixipi-lining commented on issue #430: URL: https://github.com/apache/iceberg-go/issues/430#issuecomment-2902941796 @Shreyas220 Thank you very much for your prompt reply and all your hard work! Looking forward to the release of this feature. -- This is an automated message from the Apach

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

2025-05-22 Thread via GitHub
wypoon commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2902934937 @danielcweeks are you ok with this as it stands now? This will produce 3 sets of jars, and we continue keeping the classes built against Hive 2 in the runtime jars of engines. When we ar

Re: [I] A more robust way to deprecate APIs [iceberg-python]

2025-05-22 Thread via GitHub
github-actions[bot] commented on issue #1330: URL: https://github.com/apache/iceberg-python/issues/1330#issuecomment-2902929835 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.37.0 to 2.38.0 [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #12852: URL: https://github.com/apache/iceberg/pull/12852#issuecomment-2902926791 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Hive: Support custom HMSClient [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #12712: URL: https://github.com/apache/iceberg/pull/12712#issuecomment-2902926748 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark: when doing rewrite_data_files, check for partitioning schema compatibility [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #12651: URL: https://github.com/apache/iceberg/pull/12651#issuecomment-2902926720 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Efficient Addition of New Columns in Large-Scale Feature Datasets [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on issue #11634: URL: https://github.com/apache/iceberg/issues/11634#issuecomment-2902926650 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Support server side data commit in REST [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on issue #10618: URL: https://github.com/apache/iceberg/issues/10618#issuecomment-2902926488 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Cannot connect and read glue iceberg tables with hyphens [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] commented on issue #11483: URL: https://github.com/apache/iceberg/issues/11483#issuecomment-2902926541 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Support server side data commit in REST [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] closed issue #10618: Support server side data commit in REST URL: https://github.com/apache/iceberg/issues/10618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Cannot connect and read glue iceberg tables with hyphens [iceberg]

2025-05-22 Thread via GitHub
github-actions[bot] closed issue #11483: Cannot connect and read glue iceberg tables with hyphens URL: https://github.com/apache/iceberg/issues/11483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] AWS: Skip cleanup of analytics accelerator when disabled [iceberg]

2025-05-22 Thread via GitHub
devinrsmith opened a new pull request, #13134: URL: https://github.com/apache/iceberg/pull/13134 If an S3AsyncClient has been used from S3FileIO and the analytics accelerator is disabled (is disabled by default), the S3FileIO may throw an exception during closing if the analytics accelerato

[I] AWS: Closing S3FileIO can result in ClassNotFoundException [iceberg]

2025-05-22 Thread via GitHub
devinrsmith opened a new issue, #13133: URL: https://github.com/apache/iceberg/issues/13133 ### Apache Iceberg version 1.9.0 (latest release) ### Query engine None ### Please describe the bug 🐞 If an S3AsyncClient has been used from S3FileIO and the analytic

[PR] AWS: Configure Default s3Async credentials the same as s3 [iceberg]

2025-05-22 Thread via GitHub
devinrsmith opened a new pull request, #13132: URL: https://github.com/apache/iceberg/pull/13132 Closes #13131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

[I] DefaultAwsClientFactory s3 / s3Async credentials are not equivalent [iceberg]

2025-05-22 Thread via GitHub
devinrsmith opened a new issue, #13131: URL: https://github.com/apache/iceberg/issues/13131 ### Apache Iceberg version 1.9.0 (latest release) ### Query engine None ### Please describe the bug 🐞 It's possible to configure s3FileIOProperties such that `org.ap

[PR] Spark-3.5: Add unit tests for ColumnarBatchUtil [iceberg]

2025-05-22 Thread via GitHub
anuragmantri opened a new pull request, #12275: URL: https://github.com/apache/iceberg/pull/12275 Fixes: https://github.com/apache/iceberg/issues/12054 `ColumnarBatchUtil` class was added as part of delete logic refactor https://github.com/apache/iceberg/pull/11933. This PR adds unit

Re: [I] feat: Support default values in `UpdateSchema` [iceberg-rust]

2025-05-22 Thread via GitHub
liamzwbao commented on issue #1357: URL: https://github.com/apache/iceberg-rust/issues/1357#issuecomment-2902848467 I'd like to take this one once your PR is merged @jonathanc-n. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] Tracking issues of Iceberg Rust 0.5.0 Release (May 2025) [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #1325: URL: https://github.com/apache/iceberg-rust/issues/1325#issuecomment-2902763710 Going through the [release process](https://rust.iceberg.apache.org/release.html) again for `0.5.0-rc2`. - Add new commits to changelog, #1371 - Dependency list should

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu merged PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

[PR] add new commits to changelog for 0.5.0 [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu opened a new pull request, #1371: URL: https://github.com/apache/iceberg-rust/pull/1371 ## Which issue does this PR close? - Related to #1325 ## What changes are included in this PR? Add new PRs since `0.5.0-rc1` to cut the new `0.5.0-rc2`

Re: [I] Tracking issues of Iceberg Rust 0.5.0 Release (May 2025) [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #1325: URL: https://github.com/apache/iceberg-rust/issues/1325#issuecomment-2902753665 RC1 has an issue where the `DEPENDENCIES.rust.tsv` were accidentally removed in #1345. These were added back in #1363 with the licenses fixed as well. We also found an

Re: [PR] Flink: port range distribution to v2 iceberg sink [iceberg]

2025-05-22 Thread via GitHub
stevenzwu commented on code in PR #12071: URL: https://github.com/apache/iceberg/pull/12071#discussion_r2103486896 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -645,72 +711,135 @@ private DataStream distributeDataStream(DataStream inp

Re: [PR] Flink: port range distribution to v2 iceberg sink [iceberg]

2025-05-22 Thread via GitHub
stevenzwu commented on code in PR #12071: URL: https://github.com/apache/iceberg/pull/12071#discussion_r2103486896 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -645,72 +711,135 @@ private DataStream distributeDataStream(DataStream inp

Re: [PR] Run dependency license check in release script [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu merged PR #1367: URL: https://github.com/apache/iceberg-rust/pull/1367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Add support for evolving a partition column [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko commented on PR #1334: URL: https://github.com/apache/iceberg-rust/pull/1334#issuecomment-2902740777 Thanks @kevinjqliu 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Add support for evolving a partition column [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko merged PR #1334: URL: https://github.com/apache/iceberg-rust/pull/1334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2902671382 Moved `BSL-1.0` to allowlist instead of exception. I reran both of these commands and everything looks ✅ ``` python3 ../scripts/dependencies.py check python3 ../scripts/d

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko commented on code in PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#discussion_r2103452156 ## deny.toml: ## @@ -31,8 +31,13 @@ allow = [ ] exceptions = [ + # Boost Software License Version 1.0 is allowed (Category-A): + # https://www.apache.org/legal

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2902660656 Merged #1366 and rebased, thanks fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Make `dependencies.py generate` fail on cargo-deny error [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu merged PR #1366: URL: https://github.com/apache/iceberg-rust/pull/1366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Add support for evolving a partition column [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko commented on code in PR #1334: URL: https://github.com/apache/iceberg-rust/pull/1334#discussion_r2103449335 ## crates/integration_tests/testdata/spark/provision.py: ## @@ -120,6 +120,14 @@ spark.sql("ALTER TABLE rest.default.test_promote_column ALTER COLUMN foo TYPE bigi

Re: [PR] Add support for evolving a partition column [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on code in PR #1334: URL: https://github.com/apache/iceberg-rust/pull/1334#discussion_r2103446635 ## crates/integration_tests/testdata/spark/provision.py: ## @@ -120,6 +120,14 @@ spark.sql("ALTER TABLE rest.default.test_promote_column ALTER COLUMN foo TYPE

Re: [PR] feat: add schema projection support [iceberg-cpp]

2025-05-22 Thread via GitHub
Fokko merged PR #102: URL: https://github.com/apache/iceberg-cpp/pull/102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2902618862 Great catch!! Very nice, I've added to the allow list. This PR should be good to go. I need to first merge #1366 and then rebase -- This is an automated message from the

[PR] Hive: Throw exception for when listing a non-existing namespace [iceberg]

2025-05-22 Thread via GitHub
jmelinav opened a new pull request, #13130: URL: https://github.com/apache/iceberg/pull/13130 Throw exception for when listing a non-existing namespace Fixes https://github.com/apache/iceberg/issues/12874 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Expose Avro reader to PyIceberg [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko commented on PR #1328: URL: https://github.com/apache/iceberg-rust/pull/1328#issuecomment-2902572019 Thanks for chiming in here, I've created PR https://github.com/apache/iceberg-rust/pull/1369 that implements #2. PTAL -- This is an automated message from the Apache Git Service. To

Re: [I] Cannot create a table from a pyarrow schema [iceberg-python]

2025-05-22 Thread via GitHub
DavidEscott commented on issue #2030: URL: https://github.com/apache/iceberg-python/issues/2030#issuecomment-2902490361 There seem to be two existing issues related to this, but both are closed. [#1797] and [#1100]. I've adopted the approach from 1100, and it does work, but I certainly dis

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
Fokko commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2902401243 @kevinjqliu Acronyms 🫠 According to [clipboard-win on crates.io](https://crates.io/crates/clipboard-win) it refers to the [Boost Software License 1.0](https://choosealicense.com/licens

Re: [PR] feat (glue): Add Set Table property support for Glue [iceberg-go]

2025-05-22 Thread via GitHub
maninc closed pull request #378: feat (glue): Add Set Table property support for Glue URL: https://github.com/apache/iceberg-go/pull/378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c commented on PR #9335: URL: https://github.com/apache/iceberg/pull/9335#issuecomment-2902285554 Thanks @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c opened a new pull request, #9335: URL: https://github.com/apache/iceberg/pull/9335 ### Note to reviewers - Closes https://github.com/apache/iceberg/issues/8856 - Instead of returning `ManifestFile` in `BaseAllMetadataTableScan::reachableManifests`, we return a `Pair` from

Re: [PR] Flink: port range distribution to v2 iceberg sink [iceberg]

2025-05-22 Thread via GitHub
rodmeneses commented on code in PR #12071: URL: https://github.com/apache/iceberg/pull/12071#discussion_r2103206341 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -645,72 +711,135 @@ private DataStream distributeDataStream(DataStream in

Re: [I] Cannot create a table from a pyarrow schema [iceberg-python]

2025-05-22 Thread via GitHub
DavidEscott commented on issue #2030: URL: https://github.com/apache/iceberg-python/issues/2030#issuecomment-2902134199 Not in my use-case, because I also want to specify the `PartitionSpec` for `create_table`. From what I can tell the signature of `catalog.create_table` doesn't work at pr

[PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-22 Thread via GitHub
elphastori opened a new pull request, #13129: URL: https://github.com/apache/iceberg/pull/13129 Allow listing tables, views and namespaces in rest catalog without `pageToken` query parameter when `pageSize` query parameter is included. Closes #13119 Examples: - http://localhost:81

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103108339 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103091055 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] AWS: pass storage credentials to s3 client [iceberg]

2025-05-22 Thread via GitHub
nastra commented on PR #13118: URL: https://github.com/apache/iceberg/pull/13118#issuecomment-2902047069 https://github.com/apache/iceberg/pull/12799 has been merged before I had the chance to merge your PR. Can you rebase this please so that we can get the tests in -- This is an automat

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103080451 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] [WIP] Spark4.0: RC -> official release [iceberg]

2025-05-22 Thread via GitHub
szehon-ho commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2902031997 Spark 4.0 release vote finally passed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Docs: add column descriptions for entries metadata table [iceberg]

2025-05-22 Thread via GitHub
elphastori commented on code in PR #13104: URL: https://github.com/apache/iceberg/pull/13104#discussion_r2103074320 ## docs/docs/spark-queries.md: ## @@ -301,6 +301,16 @@ SELECT * FROM prod.db.table.entries; | -- | -- | -- | -- | -- | -- | | 2 | 57897183625154 | 0 | 0 | {"con

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103068654 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103051361 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103044192 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103044192 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103006056 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c commented on PR #9335: URL: https://github.com/apache/iceberg/pull/9335#issuecomment-2901969371 @manuzhang Yes, I'll come back to it soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2901960347 Note, the license check currently fails with: ``` Checking dependencies of /Users/kevinliu/repos/iceberg-rust/crates/integrations/cli error[rejected]: failed to satisfy li

Re: [PR] [WIP] Spark4.0: RC -> official release [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2901943953 So soon! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103007012 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -81,6 +87,40 @@ record ->

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102998034 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -196,6 +196,18 @@ public static

Re: [PR] Expose Avro reader to PyIceberg [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1328: URL: https://github.com/apache/iceberg-rust/pull/1328#issuecomment-2901908181 I like #2 as well. The refactor should be less effort than scaffolding between python class and rust struct -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102997034 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -81,6 +87,40 @@ record ->

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102993338 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -89,9 +129,6 @@ public static

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2102924653 ## pyiceberg/table/inspect.py: ## @@ -678,6 +689,28 @@ def all_manifests(self) -> "pa.Table": ) return pa.concat_tables(manifests_by_sn

Re: [PR] AWS: Support multiple storage credential prefixes [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar merged PR #12799: URL: https://github.com/apache/iceberg/pull/12799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Docs: Fix Flink upsert doc on equality fields requirement [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on code in PR #13127: URL: https://github.com/apache/iceberg/pull/13127#discussion_r2102701620 ## docs/docs/flink-writes.md: ## @@ -75,7 +75,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ``` !!! info -OVERW

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2901811346 Nice i think those should cover all of them. I double checked with the rest spec * [list namespace](https://github.com/apache/iceberg/blob/91dff9886e6e6c494f6a970129f4c08487c

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2102918617 ## pyiceberg/table/inspect.py: ## @@ -678,6 +689,28 @@ def all_manifests(self) -> "pa.Table": ) return pa.concat_tables(manifests_by_sn

Re: [PR] Partition statistics metadata reading [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2033: URL: https://github.com/apache/iceberg-python/pull/2033#discussion_r2102910004 ## pyiceberg/table/statistics.py: ## @@ -29,15 +29,24 @@ class BlobMetadata(IcebergBaseModel): properties: Optional[Dict[str, str]] = None -clas

Re: [PR] Partition statistics metadata reading [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2033: URL: https://github.com/apache/iceberg-python/pull/2033#discussion_r2102909289 ## pyiceberg/table/statistics.py: ## @@ -29,15 +29,24 @@ class BlobMetadata(IcebergBaseModel): properties: Optional[Dict[str, str]] = None -clas

  1   2   3   >