Re: [I] DeleteOrphanFilesSparkAction.listDirRecursively - No FileSystem for scheme "s3" [iceberg]

2024-12-24 Thread via GitHub
raphaelauv commented on issue #10539: URL: https://github.com/apache/iceberg/issues/10539#issuecomment-2560961199 no stale -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [I] REST catalog doesn't return old history if we execute `CREATE OR REPLACE TABLE` statement [iceberg]

2024-12-24 Thread via GitHub
amogh-jahagirdar closed issue #11777: REST catalog doesn't return old history if we execute `CREATE OR REPLACE TABLE` statement URL: https://github.com/apache/iceberg/issues/11777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Core: Don't reset snapshotLog in `TableMetadata.removeRef` method [iceberg]

2024-12-24 Thread via GitHub
amogh-jahagirdar merged PR #11779: URL: https://github.com/apache/iceberg/pull/11779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] chore: improve and fix the rest example [iceberg-rust]

2024-12-24 Thread via GitHub
goldmedal opened a new pull request, #842: URL: https://github.com/apache/iceberg-rust/pull/842 # Description I'm a newer of iceberg and iceberg-rs. I'm not sure if I did it right 🤔 When I tried the example, I was confused about what I needed to prepare (e.g. rest catalog server) I

[PR] feat: support metadata table "history" [iceberg-rust]

2024-12-24 Thread via GitHub
rshkv opened a new pull request, #841: URL: https://github.com/apache/iceberg-rust/pull/841 Re #823. Extends @xxchan's #822 to add support for the "history" metadata table. (I'll rebase and update this PR once #822 merges.) The history table is the [snapshot log](https://iceberg.apac

Re: [PR] Gradle 8.12 updated the gradlew with better APP_HOME definition [iceberg]

2024-12-24 Thread via GitHub
Fokko merged PR #11869: URL: https://github.com/apache/iceberg/pull/11869 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core, Spark: Avoid deprecated methods in Guava Files [iceberg]

2024-12-24 Thread via GitHub
Fokko merged PR #11865: URL: https://github.com/apache/iceberg/pull/11865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] feat: support delete if empty for parquet writer [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME commented on PR #838: URL: https://github.com/apache/iceberg-rust/pull/838#issuecomment-2561052809 I don't add the test for this PR yet because I am not sure whether it's a reasonable design. Welcome to any suggestions. cc @liurenjie1024 @Xuanwo -- This is an automated message fr

Re: [PR] Core: Replace deprecated Schema.toString with SchemaFormatter [iceberg]

2024-12-24 Thread via GitHub
ebyhr commented on PR #11867: URL: https://github.com/apache/iceberg/pull/11867#issuecomment-2560825517 CI hit #11651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Flaky test `TestCopyOnWriteDelete > testDeleteWithSnapshotIsolation()` [iceberg]

2024-12-24 Thread via GitHub
ebyhr commented on issue #11651: URL: https://github.com/apache/iceberg/issues/11651#issuecomment-2560825773 https://github.com/apache/iceberg/actions/runs/12477951528/job/34824832444?pr=11867 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Core: Prevent dropping column which is referenced by active partition specs [iceberg]

2024-12-24 Thread via GitHub
advancedxy commented on PR #11842: URL: https://github.com/apache/iceberg/pull/11842#issuecomment-2560827832 > Just to double check, with dropping the offending column, I was assuming that you would mutate an existing spec. No, I'm not proposing to mutate an existing spec directly. Af

Re: [PR] Count rows as a metadata only operation [iceberg-python]

2024-12-24 Thread via GitHub
tusharchou commented on code in PR #1388: URL: https://github.com/apache/iceberg-python/pull/1388#discussion_r1896538967 ## pyiceberg/table/__init__.py: ## @@ -1493,6 +1496,13 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arrow())

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] feat: add s3tables catalog [iceberg-rust]

2024-12-24 Thread via GitHub
flaneur2020 commented on code in PR #807: URL: https://github.com/apache/iceberg-rust/pull/807#discussion_r1896547156 ## crates/catalog/s3tables/src/catalog.rs: ## @@ -0,0 +1,620 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Flink: Avoid RANGE mode broken chain when write parallelism changes [iceberg]

2024-12-24 Thread via GitHub
huyuanfeng2018 commented on PR #11702: URL: https://github.com/apache/iceberg/pull/11702#issuecomment-2561104437 > just a nit comment fixed. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] kafka-connector | org.apache.kafka.connect.storage.StringConverter for value not working [iceberg]

2024-12-24 Thread via GitHub
raphaelauv opened a new issue, #11871: URL: https://github.com/apache/iceberg/issues/11871 ### Apache Iceberg version 1.7.1 (latest release) ### Query engine Kafka Connect ### Please describe the bug 🐞 if i produce a message in string for key and value

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME commented on PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#issuecomment-2561039157 cc @Xuanwo @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME opened a new pull request, #837: URL: https://github.com/apache/iceberg-rust/pull/837 I find that the decimal encode for parquet statistic will be byte with var len. Not necessary with 16 len. So we need to extend it when the len smaller than 16. -- This is an automated message

[PR] feat: support delete if empty for parquet writer [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME opened a new pull request, #838: URL: https://github.com/apache/iceberg-rust/pull/838 This PR lets the user indicate to delete the file if the writer doesn't write anything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[I] Allow passing a metadata folder to `IcebergTableProviderFactory` [iceberg-rust]

2024-12-24 Thread via GitHub
gruuya opened a new issue, #840: URL: https://github.com/apache/iceberg-rust/issues/840 Currently when using the `TableProviderFactory` mechanism from DataFusion one needs to specify the full exact path to the metadata _file_ as the location, e.g. ```sql create external table invento

Re: [PR] Integrate Test Framework [iceberg-cpp]

2024-12-24 Thread via GitHub
wgtmac commented on code in PR #13: URL: https://github.com/apache/iceberg-cpp/pull/13#discussion_r1896787151 ## README.md: ## @@ -28,13 +28,14 @@ C++ implementation of [Apache Iceberg™](https://iceberg.apache.org/). ## Build -### Build and Install Core Libraries +### Buil

Re: [PR] chore: improve and fix the rest example [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo merged PR #842: URL: https://github.com/apache/iceberg-rust/pull/842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME commented on code in PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#discussion_r1896851693 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -1169,4 +1172,187 @@ mod tests { Ok(()) } + +#[tokio::test] +async fn test_

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo commented on code in PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#discussion_r1896852907 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -1169,4 +1172,187 @@ mod tests { Ok(()) } + +#[tokio::test] +async fn test_d

Re: [I] Allow passing a metadata folder to `IcebergTableProviderFactory` [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo commented on issue #840: URL: https://github.com/apache/iceberg-rust/issues/840#issuecomment-2561263257 Hi, I have a strong feeling that we need to add catalog support for datafusion. Any ideas? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] rust_decimal only supports precision less than 28 [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo commented on issue #669: URL: https://github.com/apache/iceberg-rust/issues/669#issuecomment-2561265303 Possible alternative: - https://crates.io/crates/bigdecimal - https://crates.io/crates/decimal-rs -- This is an automated message from the Apache Git Service. To respond

Re: [PR] feat: support metadata table "history" [iceberg-rust]

2024-12-24 Thread via GitHub
rshkv commented on code in PR #841: URL: https://github.com/apache/iceberg-rust/pull/841#discussion_r1896841369 ## crates/iceberg/src/metadata_scan.rs: ## @@ -0,0 +1,395 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] chore: improve and fix the rest example [iceberg-rust]

2024-12-24 Thread via GitHub
goldmedal commented on PR #842: URL: https://github.com/apache/iceberg-rust/pull/842#issuecomment-2561252015 > Thank you @goldmedal for adding this! Thanks, @Xuanwo for the review 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo commented on code in PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#discussion_r1896849650 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -1169,4 +1172,187 @@ mod tests { Ok(()) } + +#[tokio::test] +async fn test_d

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-24 Thread via GitHub
ZENOTME commented on code in PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#discussion_r1897104456 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -1169,4 +1172,187 @@ mod tests { Ok(()) } + +#[tokio::test] +async fn test_

Re: [PR] fix: revert "chore: fix typo in FileIO Schemes (#653)" [iceberg-rust]

2024-12-24 Thread via GitHub
Xuanwo commented on PR #845: URL: https://github.com/apache/iceberg-rust/pull/845#issuecomment-2561686193 We can support both of them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] The nested type identification is incorrect STRUCT>>> [iceberg]

2024-12-24 Thread via GitHub
madeirak commented on issue #11872: URL: https://github.com/apache/iceberg/issues/11872#issuecomment-2561664836 After trying, error is reported when the ARRAY and MAP of the nested fields are directly next to each other. For example, ARRAY>> or MAP>> , changing the name of field x will re

Re: [PR] ci: use officail rustsec/audit-check action [iceberg-rust]

2024-12-24 Thread via GitHub
xxchan commented on PR #843: URL: https://github.com/apache/iceberg-rust/pull/843#issuecomment-2561694106 Submitted a ticket https://issues.apache.org/jira/browse/INFRA-26399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] ci: use officail rustsec/audit-check action [iceberg-rust]

2024-12-24 Thread via GitHub
xxchan commented on PR #843: URL: https://github.com/apache/iceberg-rust/pull/843#issuecomment-2561664021 hmmm, this seems not in the approved list -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] Revert "chore: fix typo in FileIO Schemes (#653)" [iceberg-rust]

2024-12-24 Thread via GitHub
chenzl25 opened a new pull request, #845: URL: https://github.com/apache/iceberg-rust/pull/845 - This reverts commit cda4a0c595af2606e2f4076e9ef81d79d4428f4b. - The scheme of google cloud storage should be `gs` instead of `gcs` and `gs` is not a typo. `gs` is widely used by google and ope

Re: [PR] ci: add rust-cache action [iceberg-rust]

2024-12-24 Thread via GitHub
xxchan commented on PR #844: URL: https://github.com/apache/iceberg-rust/pull/844#issuecomment-2561687501 https://github.com/user-attachments/assets/4caf3b6f-41cb-44cf-980d-5b4e527fd3bd"; /> It performs much better than sccache (https://github.com/apache/iceberg-rust/pull/824#issueco

Re: [PR] feat: Bump opendal to 0.51 [iceberg-rust]

2024-12-24 Thread via GitHub
liurenjie1024 merged PR #839: URL: https://github.com/apache/iceberg-rust/pull/839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Integrate Test Framework [iceberg-cpp]

2024-12-24 Thread via GitHub
zhjwpku commented on code in PR #13: URL: https://github.com/apache/iceberg-cpp/pull/13#discussion_r1897045482 ## test/CMakeLists.txt: ## @@ -14,3 +14,7 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations #

Re: [I] Tracking issues of iceberg rust v0.4.0 Release [iceberg-rust]

2024-12-24 Thread via GitHub
sungwy closed issue #739: Tracking issues of iceberg rust v0.4.0 Release URL: https://github.com/apache/iceberg-rust/issues/739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Allow passing a metadata folder to `IcebergTableProviderFactory` [iceberg-rust]

2024-12-24 Thread via GitHub
gruuya commented on issue #840: URL: https://github.com/apache/iceberg-rust/issues/840#issuecomment-2561395335 > we need to add catalog support for datafusion Hmm, I think that is adequately supported already, no? https://github.com/apache/iceberg-rust/blob/5400bbee9697189bb95646f5

Re: [PR] feat: Support Bucket and Truncate transforms on write [iceberg-python]

2024-12-24 Thread via GitHub
kevinjqliu commented on code in PR #1345: URL: https://github.com/apache/iceberg-python/pull/1345#discussion_r1896992186 ## tests/integration/test_writes/test_partitioned_writes.py: ## @@ -719,50 +719,105 @@ def test_invalid_arguments(spark: SparkSession, session_catalog: Catal

[PR] Rest catalog integration testing [iceberg-python]

2024-12-24 Thread via GitHub
AhmedNader42 opened a new pull request, #1469: URL: https://github.com/apache/iceberg-python/pull/1469 This PR resolves #1439 by adding integration tests for the REST Catalog. Functionality testing against the server can be simulated to a certain degree, but some checks are very hard

Re: [I] (AWS Lake Formation shared resources) Iceberg tables in AWS Glue catalog has a different root namespace than the original [iceberg-python]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #845: URL: https://github.com/apache/iceberg-python/issues/845#issuecomment-2561503005 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] Table scan using functional filters [iceberg-python]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #170: URL: https://github.com/apache/iceberg-python/issues/170#issuecomment-2561503025 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] Table scan using functional filters [iceberg-python]

2024-12-24 Thread via GitHub
github-actions[bot] closed issue #170: Table scan using functional filters URL: https://github.com/apache/iceberg-python/issues/170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] Improve Iceberg REST S3SignRequest to include affected object [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] closed issue #10486: Improve Iceberg REST S3SignRequest to include affected object URL: https://github.com/apache/iceberg/issues/10486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Consolidate Error Definitions in REST Spec [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #10571: URL: https://github.com/apache/iceberg/issues/10571#issuecomment-2561502112 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] AWS: Creating a Glue table with Lake Formation enabled fails [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #10226: URL: https://github.com/apache/iceberg/issues/10226#issuecomment-2561502071 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Improve Iceberg REST S3SignRequest to include affected object [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #10486: URL: https://github.com/apache/iceberg/issues/10486#issuecomment-2561502090 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add missing error codes to REST spec [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #10570: URL: https://github.com/apache/iceberg/issues/10570#issuecomment-2561502108 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Iceberg tables creation using Spark2.4 [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] commented on issue #10479: URL: https://github.com/apache/iceberg/issues/10479#issuecomment-2561502081 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Iceberg tables creation using Spark2.4 [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] closed issue #10479: Iceberg tables creation using Spark2.4 URL: https://github.com/apache/iceberg/issues/10479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] feat: Support Bucket and Truncate transforms on write [iceberg-python]

2024-12-24 Thread via GitHub
sungwy commented on code in PR #1345: URL: https://github.com/apache/iceberg-python/pull/1345#discussion_r1897010250 ## tests/test_transforms.py: ## @@ -1563,3 +1561,43 @@ def test_ymd_pyarrow_transforms( else: with pytest.raises(ValueError): transfor

Re: [I] AWS: Creating a Glue table with Lake Formation enabled fails [iceberg]

2024-12-24 Thread via GitHub
github-actions[bot] closed issue #10226: AWS: Creating a Glue table with Lake Formation enabled fails URL: https://github.com/apache/iceberg/issues/10226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]

2024-12-24 Thread via GitHub
kevinjqliu commented on code in PR #11845: URL: https://github.com/apache/iceberg/pull/11845#discussion_r1896903736 ## site/docs/spark-quickstart.md: ## @@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name. df = spark.table("demo.nyc.taxis").show()

[I] Support Adding File Metadata Directly [iceberg-python]

2024-12-24 Thread via GitHub
subthedubdub opened a new issue, #1470: URL: https://github.com/apache/iceberg-python/issues/1470 ### Feature Request / Improvement Support a table transaction where the user can directly supply the file metadata, similar to the [Java Interface](https://javadoc.io/doc/org.apache.iceb