Re: [PR] Flink: Support source watermark for flink sql windows [iceberg]

pvary commented on PR #12191: URL: https://github.com/apache/iceberg/pull/12191#issuecomment-2644565965 Nit: Renamed the PR to match the general patterns -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] support create table like in flink catalog [iceberg]

pvary commented on code in PR #12199: URL: https://github.com/apache/iceberg/pull/12199#discussion_r1947503325 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogTable.java: ## @@ -188,6 +188,23 @@ public void testCreateTableLike() throws TableNotExistE

Re: [PR] support create table like in flink catalog [iceberg]

pvary commented on code in PR #12199: URL: https://github.com/apache/iceberg/pull/12199#discussion_r1947502775 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableNam

Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]

mapleFU commented on code in PR #11966: URL: https://github.com/apache/iceberg/pull/11966#discussion_r1947502499 ## docs/docs/amoro.md: ## @@ -0,0 +1,89 @@ +--- +title: "Apache Amoro" +--- + + +# Apache Amoro With Iceberg + +**[Apache Amoro(incubating)](https://amoro.apache.org)

Re: [PR] support create table like in flink catalog [iceberg]

pvary commented on code in PR #12199: URL: https://github.com/apache/iceberg/pull/12199#discussion_r1947502501 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -332,7 +335,15 @@ public List listTables(String databaseName) public CatalogTabl

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

jbonofre commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2644558524 > There are 2 different KC runtimes, one that includes an HMS client and one that doesn't, so with the current license/notice I took a conservative approach and included Hive, so not s

Re: [PR] support create table like in flink catalog [iceberg]

pvary commented on code in PR #12199: URL: https://github.com/apache/iceberg/pull/12199#discussion_r1947500675 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogTable.java: ## @@ -188,6 +188,23 @@ public void testCreateTableLike() throws TableNotExistE

Re: [PR] support create table like in flink catalog [iceberg]

pvary commented on code in PR #12199: URL: https://github.com/apache/iceberg/pull/12199#discussion_r1947500200 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -332,7 +335,15 @@ public List listTables(String databaseName) public CatalogTabl

Re: [PR] feat: support strict projection [iceberg-rust]

Xuanwo commented on code in PR #946: URL: https://github.com/apache/iceberg-rust/pull/946#discussion_r1947494581 ## crates/iceberg/src/expr/visitors/strict_projection.rs: ## @@ -0,0 +1,3231 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: support strict projection [iceberg-rust]

ZENOTME commented on code in PR #946: URL: https://github.com/apache/iceberg-rust/pull/946#discussion_r1947490272 ## crates/iceberg/src/expr/visitors/strict_projection.rs: ## @@ -0,0 +1,3231 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributo

Re: [PR] feat: support strict projection [iceberg-rust]

ZENOTME commented on code in PR #946: URL: https://github.com/apache/iceberg-rust/pull/946#discussion_r1947484109 ## crates/iceberg/src/spec/transform.rs: ## @@ -296,6 +297,165 @@ impl Transform { } } +/// Strictly projects a given predicate according to the

Re: [PR] Add pyiceberg DataFusion e2e test [iceberg-rust]

Xuanwo commented on PR #825: URL: https://github.com/apache/iceberg-rust/pull/825#issuecomment-2644503595 Hi, sorry for the late merge call. Would you like to help fix the build? @gruuya -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat(datafusion): Expose DataFusion statistics on an IcebergTableScan [iceberg-rust]

Xuanwo commented on code in PR #880: URL: https://github.com/apache/iceberg-rust/pull/880#discussion_r1947475638 ## crates/integration_tests/Cargo.toml: ## @@ -34,5 +34,6 @@ iceberg-catalog-rest = { workspace = true } iceberg-datafusion = { workspace = true } iceberg_test_util

Re: [I] Add support to FileIO for Azure and GCS [iceberg-go]

subkanthi commented on issue #65: URL: https://github.com/apache/iceberg-go/issues/65#issuecomment-2644501163 hi @zeroshade , is GCS also covered in this PR, if not happy to start working on it. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add pyiceberg DataFusion e2e test [iceberg-rust]

Xuanwo commented on code in PR #825: URL: https://github.com/apache/iceberg-rust/pull/825#discussion_r1947468058 ## crates/integration_tests/tests/datafusion.rs: ## @@ -0,0 +1,150 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] feat: Make some REST methods public [iceberg-rust]

Xuanwo commented on PR #922: URL: https://github.com/apache/iceberg-rust/pull/922#issuecomment-2644500284 > * Makes `ErrorResponse`, `OK`, and `HttpClient` public to support escape-hatching for REST catalogs that don't 100% support the iceberg catalog response formats. Hi, I'm a bit

Re: [I] Manifests table scan should return iceberg schema rather arrow schema [iceberg-rust]

Xuanwo closed issue #868: Manifests table scan should return iceberg schema rather arrow schema URL: https://github.com/apache/iceberg-rust/issues/868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] fix(metadata): export iceberg schema in manifests table [iceberg-rust]

Xuanwo merged PR #871: URL: https://github.com/apache/iceberg-rust/pull/871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Manifests table scan should return iceberg schema rather arrow schema [iceberg-rust]

Xuanwo closed issue #868: Manifests table scan should return iceberg schema rather arrow schema URL: https://github.com/apache/iceberg-rust/issues/868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.80.0 [iceberg-rust]

dependabot[bot] commented on PR #928: URL: https://github.com/apache/iceberg-rust/pull/928#issuecomment-2644495361 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.80.0 [iceberg-rust]

Xuanwo commented on PR #928: URL: https://github.com/apache/iceberg-rust/pull/928#issuecomment-2644495355 Thank you @kevinjqliu for taking care of this. We don't need to merge this since they are compatible. Users can upgrade to version 1.80 on their own. -- This is an auto

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.80.0 [iceberg-rust]

Xuanwo closed pull request #928: chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.80.0 URL: https://github.com/apache/iceberg-rust/pull/928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] fix: gurantee the deserialize order of struct is same as the struct type [iceberg-rust]

Xuanwo commented on PR #795: URL: https://github.com/apache/iceberg-rust/pull/795#issuecomment-2644493194 > Looks we already have a related comment. We can close this PR. Great. Thank you for researching on this. -- This is an automated message from the Apache Git Service. To respon

Re: [PR] fix: gurantee the deserialize order of struct is same as the struct type [iceberg-rust]

Xuanwo closed pull request #795: fix: gurantee the deserialize order of struct is same as the struct type URL: https://github.com/apache/iceberg-rust/pull/795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] fix: gurantee the deserialize order of struct is same as the struct type [iceberg-rust]

ZENOTME commented on PR #795: URL: https://github.com/apache/iceberg-rust/pull/795#issuecomment-2644493012 > > According to this, it works if we only use the RawLiteralEnum for avro. Maybe we can just add comment for this and rollback this commit? > > Yes, I believe we only need to ad

Re: [PR] fix: Make s3tables catalog public [iceberg-rust]

Xuanwo merged PR #918: URL: https://github.com/apache/iceberg-rust/pull/918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] feat(puffin): Add PuffinReader [iceberg-rust]

Xuanwo merged PR #892: URL: https://github.com/apache/iceberg-rust/pull/892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] chore: use shared containers for integration tests [iceberg-rust]

Xuanwo merged PR #924: URL: https://github.com/apache/iceberg-rust/pull/924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] feat: support strict projection [iceberg-rust]

Xuanwo commented on code in PR #946: URL: https://github.com/apache/iceberg-rust/pull/946#discussion_r1947461987 ## crates/iceberg/src/spec/transform.rs: ## @@ -296,6 +297,165 @@ impl Transform { } } +/// Strictly projects a given predicate according to the t

Re: [I] S3 Connection Error with Glue Catalog [iceberg-rust]

Xuanwo commented on issue #940: URL: https://github.com/apache/iceberg-rust/issues/940#issuecomment-2644489661 Thank you @omerhadari, closed by https://github.com/apache/iceberg-rust/pull/947 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] fix: gurantee the deserialize order of struct is same as the struct type [iceberg-rust]

Xuanwo commented on PR #795: URL: https://github.com/apache/iceberg-rust/pull/795#issuecomment-2644490002 > According to this, it works if we only use the RawLiteralEnum for avro. Maybe we can just add comment for this and rollback this commit? Yes, I believe we only need to add comme

Re: [I] S3 Connection Error with Glue Catalog [iceberg-rust]

Xuanwo closed issue #940: S3 Connection Error with Glue Catalog URL: https://github.com/apache/iceberg-rust/issues/940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] feat(glue): use the same props for creating aws sdk and for FileIO [iceberg-rust]

Xuanwo merged PR #947: URL: https://github.com/apache/iceberg-rust/pull/947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spark: support statistics files in RewriteTablePath [iceberg]

szehon-ho commented on code in PR #11929: URL: https://github.com/apache/iceberg/pull/11929#discussion_r1947406948 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -350,18 +352,47 @@ private RewriteResult rewriteVersionF

Re: [I] [DISCUSS] Columnar data protocol: Arrow or implement a new one? [iceberg-cpp]

wgtmac commented on issue #33: URL: https://github.com/apache/iceberg-cpp/issues/33#issuecomment-2644431805 @JohanMabille Thanks for the reply! I just read your [post](https://johan-mabille.medium.com/sparrow-1f23817f6696) and will definitely try it out! -- This is an automated message f

Re: [PR] Spark: support statistics files in RewriteTablePath [iceberg]

szehon-ho commented on code in PR #11929: URL: https://github.com/apache/iceberg/pull/11929#discussion_r1947405782 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -350,18 +352,47 @@ private RewriteResult rewriteVersionF

Re: [PR] Spark: support statistics files in RewriteTablePath [iceberg]

szehon-ho commented on code in PR #11929: URL: https://github.com/apache/iceberg/pull/11929#discussion_r1947405782 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -350,18 +352,47 @@ private RewriteResult rewriteVersionF

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

szehon-ho commented on PR #12172: URL: https://github.com/apache/iceberg/pull/12172#issuecomment-2644399196 Thanks a lot @barronfuentes and also @dramaticlly for evaluating and reviewing it! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

szehon-ho merged PR #12172: URL: https://github.com/apache/iceberg/pull/12172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

szehon-ho commented on PR #12172: URL: https://github.com/apache/iceberg/pull/12172#issuecomment-2644398600 This looks good, it looks like a similar issue as https://github.com/apache/iceberg/pull/12006 but for metadata files. -- This is an automated message from the Apache Git Service. T

Re: [I] Iceberg SDK failed to clean up files when table has multiple references with different retention time [iceberg]

MavsLee commented on issue #12200: URL: https://github.com/apache/iceberg/issues/12200#issuecomment-2644396979 I'm working on a draft PR fix and unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[I] Iceberg SDK failed to clean up files when table has multiple references with different retention time [iceberg]

MavsLee opened a new issue, #12200: URL: https://github.com/apache/iceberg/issues/12200 ### Apache Iceberg version 1.7.1 (latest release) ### Query engine Other ### Please describe the bug 🐞 When using Iceberg Java SDK `org.apache.iceberg.RemoveSnapshots`to

Re: [I] [Feature] Provide Nightly Build to PyPi [iceberg-python]

kevinjqliu commented on issue #872: URL: https://github.com/apache/iceberg-python/issues/872#issuecomment-2644392680 First cron scheduled run is ✅ https://github.com/apache/iceberg-python/actions/runs/13210296696 https://test.pypi.org/project/pyiceberg/0.9.0.dev20250208002427/ -- Th

Re: [PR] support source watermark for flink sql windows [iceberg]

swapna267 commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1947376035 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceSql.java: ## @@ -162,4 +173,74 @@ public void testWatermarkOptionsDescending() t

Re: [PR] support source watermark for flink sql windows [iceberg]

swapna267 commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1947355037 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/SqlBase.java: ## @@ -86,7 +86,7 @@ protected void dropDatabase(String database, boolean ifExists) {

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

gabeiglio commented on PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#issuecomment-2644363377 @kevinjqliu IIUC removing the schema casting will allow pyarrow scanner to infer by itself if it needs or not large types? So it is basically a matter of changing the assertions

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

github-actions[bot] commented on PR #11525: URL: https://github.com/apache/iceberg/pull/11525#issuecomment-2644360143 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] support source watermark for flink sql windows [iceberg]

swapna267 commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1947348896 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceSql.java: ## @@ -162,4 +173,74 @@ public void testWatermarkOptionsDescending() t

Re: [PR] Spark: support statistics files in RewriteTablePath [iceberg]

dramaticlly commented on PR #11929: URL: https://github.com/apache/iceberg/pull/11929#issuecomment-2644361070 Not stale, will rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Spark: support statistics files in RewriteTablePath [iceberg]

github-actions[bot] commented on PR #11929: URL: https://github.com/apache/iceberg/pull/11929#issuecomment-2644360203 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]

swapna267 commented on code in PR #12116: URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTabl

Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]

swapna267 commented on code in PR #12116: URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTabl

Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]

swapna267 commented on code in PR #12116: URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTabl

Re: [PR] Build: Bump mkdocstrings-python from 1.14.5 to 1.14.6 [iceberg-python]

kevinjqliu merged PR #1624: URL: https://github.com/apache/iceberg-python/pull/1624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]

swapna267 commented on code in PR #12116: URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947295585 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTabl

Re: [PR] Build: Bump mkdocstrings-python from 1.14.5 to 1.14.6 [iceberg-python]

kevinjqliu commented on PR #1624: URL: https://github.com/apache/iceberg-python/pull/1624#issuecomment-2644276388 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump mkdocs-material from 9.6.2 to 9.6.3 [iceberg-python]

kevinjqliu merged PR #1625: URL: https://github.com/apache/iceberg-python/pull/1625 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

[PR] support create table like in flink catalog [iceberg]

swapna267 opened a new pull request, #12199: URL: https://github.com/apache/iceberg/pull/12199 Creation of dynamic Iceberg table in Flink Catalog using the underlying physical Iceberg table using LIKE clause. Currently (without the changes in PR), create table in flink catalog works

[PR] Build: Bump mkdocs-material from 9.6.2 to 9.6.3 [iceberg-python]

dependabot[bot] opened a new pull request, #1625: URL: https://github.com/apache/iceberg-python/pull/1625 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.2 to 9.6.3. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mk

[PR] Build: Bump mkdocstrings-python from 1.14.5 to 1.14.6 [iceberg-python]

dependabot[bot] opened a new pull request, #1624: URL: https://github.com/apache/iceberg-python/pull/1624 Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.14.5 to 1.14.6. Release notes Sourced from https://github.com/mkdocstrings/python/releases";>mkdocstr

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

tscottcoombes1 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2644227791 Thanks @mattmartin14 appreciate you keep plugging away. I think it looks really good. I've left a couple more comments (sorry). IMO only the pyproject.toml is a blocker, eve

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

tscottcoombes1 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1947262347 ## tests/table/test_upsert.py: ## @@ -0,0 +1,327 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreement

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

tscottcoombes1 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1947257232 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

tscottcoombes1 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1947249998 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,146 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Remove old metadata [iceberg-python]

kaushiksrini commented on PR #1607: URL: https://github.com/apache/iceberg-python/pull/1607#issuecomment-2644180446 @Fokko thanks! used context managers and added to the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] feat: implement InMemoryCatalog as a subclass of SqlCatalog [iceberg-python]

hussein-awala commented on code in PR #1140: URL: https://github.com/apache/iceberg-python/pull/1140#discussion_r1947182388 ## tests/catalog/test_memory.py: ## @@ -278,17 +62,20 @@ def catalog(tmp_path: PosixPath) -> InMemoryCatalog: ) TEST_TABLE_PARTITION_SPEC = PartitionSpec

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

tscottcoombes1 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1947207894 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,158 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] [infra] nightly pypi build for `pyiceberg_core` [iceberg-rust]

kevinjqliu commented on code in PR #948: URL: https://github.com/apache/iceberg-rust/pull/948#discussion_r1947190670 ## .github/workflows/release_python_nightly.yml: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [I] [feature] Add support for `write.data.path` and `write.metadata.path` [iceberg-python]

geruh commented on issue #1492: URL: https://github.com/apache/iceberg-python/issues/1492#issuecomment-2644109473 I'd like to pick up the `write.metadata.path` impl if no one has already! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] [feature] Add support for `write.data.path` and `write.metadata.path` [iceberg-python]

kevinjqliu commented on issue #1492: URL: https://github.com/apache/iceberg-python/issues/1492#issuecomment-2644116402 sure @geruh thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

dramaticlly commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1947170590 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() throws

Re: [PR] feat: implement InMemoryCatalog as a subclass of SqlCatalog [iceberg-python]

hussein-awala commented on PR #1140: URL: https://github.com/apache/iceberg-python/pull/1140#issuecomment-2644093341 > hey @hussein-awala would you like to make the above changes on docs? This PR is almost ready! yes, I will make it ready ASAP -- This is an automated message from t

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

barronfuentes commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1947141196 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() thro

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

barronfuentes commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1947143198 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() thro

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

barronfuentes commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1947140415 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() thro

Re: [I] Delete Files in Table Scans [iceberg-rust]

sdd commented on issue #630: URL: https://github.com/apache/iceberg-rust/issues/630#issuecomment-2644037450 Hi all. I'm resurrecting this issue now that @Fokko has kindly helped get the first part of this over the line by reviewing and merging https://github.com/apache/iceberg-rust/pull/652

Re: [PR] Core: add variant type support [iceberg]

aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1947067468 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -370,4 +370,17 @@ public void testFieldDocsArePreserved() { Lists.newArrayLis

Re: [PR] Spec: Support geo type [iceberg]

redblackcoder commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1947105947 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

[PR] Auth manager 6 final2 [iceberg]

adutra opened a new pull request, #12197: URL: https://github.com/apache/iceberg/pull/12197 6th and last PR for the Auth Manager API. Previous ones: * #11995 * #11992 * #11844 * #11809 * #11769 Once this PR is merged, the AuthManager API becomes effective. Su

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distribution [iceberg]

jbonofre commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2643831588 @bryanck yes. I saw that. Both distributions use the same LICENSE/NOTICE so it won't be good either way 😆 Let me update with the largest common denominator. I'm checking. -- This

Re: [PR] Spec: Support geo type [iceberg]

desruisseaux commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1947060354 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

Re: [I] Support Nessie catalog [iceberg-python]

gmweaver commented on issue #19: URL: https://github.com/apache/iceberg-python/issues/19#issuecomment-2643732451 I confirmed that it is using S3 configs set in the catalog and not those configured on the Nessie Server, the following works: ``` catalog = load_catalog( "nessie

Re: [I] Support Nessie catalog [iceberg-python]

gmweaver commented on issue #19: URL: https://github.com/apache/iceberg-python/issues/19#issuecomment-2643719088 I ran into the same issue as @cee-shubham and my initial guess is that S3 bucket authentication is not using the S3 keys configured on the server and is instead trying to use loc

Re: [I] [REST Catalog] OAuth 2 grant type "refresh_token" not implemented [iceberg]

nika-qubit commented on issue #12196: URL: https://github.com/apache/iceberg/issues/12196#issuecomment-2643719831 Thanks, @danielcweeks. Yes, I was confused by the "refreshToken" method naming. It's actually doing token-exchange and the oauth server that I'm interacting with doesn't support

Re: [I] [REST Catalog] OAuth 2 grant type "refresh_token" not implemented [iceberg]

danielcweeks commented on issue #12196: URL: https://github.com/apache/iceberg/issues/12196#issuecomment-2643707900 @nika-qubit I think you're confusing two separate things here. The token exchange flow is used by the current implementation to provide a way to get a new token using a valid

Re: [PR] Spec: Support geo type [iceberg]

redblackcoder commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1946995973 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

Re: [PR] Spec: Support geo type [iceberg]

redblackcoder commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1946993918 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

Re: [PR] Spec: Support geo type [iceberg]

redblackcoder commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1946979428 ## format/spec.md: ## @@ -623,6 +649,8 @@ Notes: 5. The `content_offset` and `content_size_in_bytes` fields are used to reference a specific blob for direct a

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

dramaticlly commented on PR #12172: URL: https://github.com/apache/iceberg/pull/12172#issuecomment-2643670878 FYI @flyrain @amogh-jahagirdar I think we might want take this fix as part of 1.8 release if possible. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Spec: Support geo type [iceberg]

jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1946970567 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

Re: [PR] Spec: Support geo type [iceberg]

redblackcoder commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1946965382 ## format/spec.md: ## @@ -228,6 +230,30 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +[1001]:

[I] Writing map types [iceberg-python]

b-phi opened a new issue, #1623: URL: https://github.com/apache/iceberg-python/issues/1623 ### Question I'm not sure if this is a bug or I'm just not structuring the data correctly, I couldn't find any examples for writing maps. Given a table with a simple schema with a map fie

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distribution [iceberg]

bryanck commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2643656595 There are 2 different KC runtimes, one that includes an HMS client and one that doesn't, so with the current license/notice I took a conservative approach and included Hive, so not sure

Re: [PR] Add base headers in properties to signer_headers [iceberg-python]

tom-s-powell commented on PR #1610: URL: https://github.com/apache/iceberg-python/pull/1610#issuecomment-2643630453 Could I check what sort of documentation you'd want to see? > To configure custom headers in RESTCatalog, include them in the catalog properties with the prefix `header.`.

[PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distribution [iceberg]

jbonofre opened a new pull request, #12195: URL: https://github.com/apache/iceberg/pull/12195 - Update version - Cleanup jars not found in the `lib` folder of the distribution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

dramaticlly commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1945875485 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() throws

Re: [PR] Remove old metadata [iceberg-python]

Fokko commented on code in PR #1607: URL: https://github.com/apache/iceberg-python/pull/1607#discussion_r1946894264 ## tests/catalog/test_sql.py: ## @@ -1613,3 +1614,60 @@ def test_merge_manifests_local_file_system(catalog: SqlCatalog, arrow_table_with tbl.append(arrow

[PR] REST: Extended header support for RESTClient implementations [iceberg]

gaborkaszab opened a new pull request, #12194: URL: https://github.com/apache/iceberg/pull/12194 The freshness-aware table loading requires some additional support for HTTP headers: - Response headers for get and post requests - Input headers for get request Extended the RESTCli

Re: [PR] Core: add variant type support [iceberg]

aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1946870340 ## core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java: ## @@ -56,6 +56,10 @@ class BuildAvroProjection extends AvroCustomOrderSchemaVisitor names,

Re: [I] software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. [iceberg]

timoha commented on issue #11997: URL: https://github.com/apache/iceberg/issues/11997#issuecomment-2643491543 Per #9785, we've been able to make it work for pyspark after upgrading to `org.apache.hadoop:hadoop-aws:3.4.1`, `org.apache.hadoop:hadoop-common:3.4.1`, `org.apache.iceberg:iceberg-

  1   2   >