[I] i meet a question. [iceberg]

2024-12-29 Thread via GitHub
hashmapybx opened a new issue, #11889: URL: https://github.com/apache/iceberg/issues/11889 ### Query engine spark-sql ### Question 命令行根据时间戳做时间旅行查询报错 spark-sql (cjh_test)> select * from cjh_test.datalake_iceberg_table timestamp as of '2024-12-30 10:28:57.321'; Inval

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-29 Thread via GitHub
jiakai-li commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899180251 ## pyiceberg/io/pyarrow.py: ## @@ -362,6 +362,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste "region"

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-29 Thread via GitHub
jiakai-li commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899180251 ## pyiceberg/io/pyarrow.py: ## @@ -362,6 +362,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste "region"

Re: [PR] Core: Fix typo in HadoopTableOperations [iceberg]

2024-12-29 Thread via GitHub
okumin commented on PR #11880: URL: https://github.com/apache/iceberg/pull/11880#issuecomment-2564867843 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] fix: parse var len of decimal for parquet statistic [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on PR #837: URL: https://github.com/apache/iceberg-rust/pull/837#issuecomment-2564967032 Hi @liurenjie1024 @Xuanwo , is there other question about this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add orbstack guide [iceberg-rust]

2024-12-29 Thread via GitHub
lewiszlw commented on code in PR #856: URL: https://github.com/apache/iceberg-rust/pull/856#discussion_r1899296910 ## docs/contributing/orbstack.md: ## @@ -0,0 +1,39 @@ + + +# OrbStack as a docker alternative on macOS +1. Install OrbStack by downloading [installer](https://orbst

Re: [I] i meet a question,time travel has some error to the query of timestamp [iceberg]

2024-12-29 Thread via GitHub
Fokko commented on issue #11889: URL: https://github.com/apache/iceberg/issues/11889#issuecomment-2565070938 Thanks @hashmapybx for raising this question, unfortunately, the information that you provide is very little. At first glance, the SQL seems to be correct. What kind of version of Sp

Re: [I] i meet a question,time travel has some error to the query of timestamp [iceberg]

2024-12-29 Thread via GitHub
hashmapybx commented on issue #11889: URL: https://github.com/apache/iceberg/issues/11889#issuecomment-2565084250 Additional Notes: https://github.com/user-attachments/assets/efbf317f-db87-4c65-84eb-2fe4d2da955b"; /> https://github.com/user-attachments/assets/96cabcc5-7f5e-412f-955b-d3

Re: [PR] feat: support metadata table "snapshots" [iceberg-rust]

2024-12-29 Thread via GitHub
rshkv commented on PR #822: URL: https://github.com/apache/iceberg-rust/pull/822#issuecomment-2564747091 Yeah, totally! > BTW I think we can also change the details like signatures and fields on demand later. Yeah that's what I was worried about. Sg. 👍🏻 -- This is an autom

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-12-29 Thread via GitHub
zzeekk commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2564779267 how can i reference parameters in an ArgoCD template? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] WIP: feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2024-12-29 Thread via GitHub
felixscherz commented on PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#issuecomment-2564764850 I think you can now review this PR if you have time @kevinjqliu :) The biggest issue for now will be that testing is only possible against AWS itself since moto does not supp

[PR] chore(deps): Bump aws-sdk-s3tables from 1.1.0 to 1.2.0 [iceberg-rust]

2024-12-29 Thread via GitHub
dependabot[bot] opened a new pull request, #858: URL: https://github.com/apache/iceberg-rust/pull/858 Bumps [aws-sdk-s3tables](https://github.com/awslabs/aws-sdk-rust) from 1.1.0 to 1.2.0. Commits See full diff in https://github.com/awslabs/aws-sdk-rust/commits";>compare view

[PR] chore(deps): Bump reqwest from 0.12.10 to 0.12.11 [iceberg-rust]

2024-12-29 Thread via GitHub
dependabot[bot] opened a new pull request, #859: URL: https://github.com/apache/iceberg-rust/pull/859 Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.10 to 0.12.11. Release notes Sourced from https://github.com/seanmonstar/reqwest/releases";>reqwest's releases.

[PR] chore(deps): Bump serde from 1.0.216 to 1.0.217 [iceberg-rust]

2024-12-29 Thread via GitHub
dependabot[bot] opened a new pull request, #860: URL: https://github.com/apache/iceberg-rust/pull/860 Bumps [serde](https://github.com/serde-rs/serde) from 1.0.216 to 1.0.217. Release notes Sourced from https://github.com/serde-rs/serde/releases";>serde's releases. v1.0.217

[PR] feat: Support metadata table "Manifests" [iceberg-rust]

2024-12-29 Thread via GitHub
flaneur2020 opened a new pull request, #861: URL: https://github.com/apache/iceberg-rust/pull/861 Re https://github.com/apache/iceberg-rust/issues/823. Extends @xxchan's https://github.com/apache/iceberg-rust/pull/822 to add support for the [Manifests](https://github.com/apache/iceberg/blob

Re: [PR] feat: support metadata table "snapshots" [iceberg-rust]

2024-12-29 Thread via GitHub
flaneur2020 commented on code in PR #822: URL: https://github.com/apache/iceberg-rust/pull/822#discussion_r1899100103 ## crates/iceberg/src/metadata_scan.rs: ## @@ -0,0 +1,265 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.71.0 to 1.71.1 [iceberg-go]

2024-12-29 Thread via GitHub
dependabot[bot] opened a new pull request, #239: URL: https://github.com/apache/iceberg-go/pull/239 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.71.0 to 1.71.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/5a964704cb2640ed

Re: [PR] feat: search current working directory for config file [iceberg-python]

2024-12-29 Thread via GitHub
IndexSeek commented on PR #1464: URL: https://github.com/apache/iceberg-python/pull/1464#issuecomment-2564799798 > I've left one comment, but apart from that, it looks good to me 👍 Thank you! I applied that change! ✅ -- This is an automated message from the Apache Git Service. To r

Re: [PR] Add orbstack guide [iceberg-rust]

2024-12-29 Thread via GitHub
kevinjqliu commented on code in PR #856: URL: https://github.com/apache/iceberg-rust/pull/856#discussion_r1899255390 ## docs/contributing/orbstack.md: ## @@ -0,0 +1,39 @@ + + +# OrbStack as a docker alternative on macOS +1. Install OrbStack by downloading [installer](https://orb

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-12-29 Thread via GitHub
huaxingao commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1899171737 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnVectorBuilder.java: ## @@ -46,8 +46,8 @@ public ColumnVector build(VectorHolder holde

Re: [PR] Rest catalog integration testing [iceberg-python]

2024-12-29 Thread via GitHub
AhmedNader42 commented on PR #1469: URL: https://github.com/apache/iceberg-python/pull/1469#issuecomment-2564818223 Hello @kevinjqliu, Please can we have a review for this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Add orbstack guide [iceberg-rust]

2024-12-29 Thread via GitHub
liurenjie1024 merged PR #856: URL: https://github.com/apache/iceberg-rust/pull/856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Fix typo in HadoopTableOperations [iceberg]

2024-12-29 Thread via GitHub
Fokko merged PR #11880: URL: https://github.com/apache/iceberg/pull/11880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core: Fix typo in HadoopTableOperations [iceberg]

2024-12-29 Thread via GitHub
Fokko commented on PR #11880: URL: https://github.com/apache/iceberg/pull/11880#issuecomment-2564835789 Thanks for fixing this @okumin and thanks for the review @ebyhr 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump mkdocs-awesome-pages-plugin from 2.10.0 to 2.10.1 [iceberg]

2024-12-29 Thread via GitHub
Fokko merged PR #11885: URL: https://github.com/apache/iceberg/pull/11885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.39 to 2.29.43 [iceberg]

2024-12-29 Thread via GitHub
Fokko merged PR #11886: URL: https://github.com/apache/iceberg/pull/11886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Feat/update sort order [iceberg-python]

2024-12-29 Thread via GitHub
JasperHG90 commented on PR #1465: URL: https://github.com/apache/iceberg-python/pull/1465#issuecomment-2564834930 Thanks @Fokko ! Hadn't realized that I opened the PR against this repo already. This is just scaffolding code. I will update it the coming week with the actual logic. -- This

Re: [PR] chore(deps): Bump reqwest from 0.12.10 to 0.12.11 [iceberg-rust]

2024-12-29 Thread via GitHub
Fokko merged PR #859: URL: https://github.com/apache/iceberg-rust/pull/859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] chore(deps): Bump serde from 1.0.216 to 1.0.217 [iceberg-rust]

2024-12-29 Thread via GitHub
Fokko merged PR #860: URL: https://github.com/apache/iceberg-rust/pull/860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] chore(deps): Bump aws-sdk-s3tables from 1.1.0 to 1.2.0 [iceberg-rust]

2024-12-29 Thread via GitHub
Fokko merged PR #858: URL: https://github.com/apache/iceberg-rust/pull/858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Add orbstack guide [iceberg-rust]

2024-12-29 Thread via GitHub
lewiszlw commented on code in PR #856: URL: https://github.com/apache/iceberg-rust/pull/856#discussion_r1899234586 ## CONTRIBUTING.md: ## @@ -116,7 +116,7 @@ cargo 1.69.0 (6e9a83356 2023-04-12) Currently, iceberg-rust uses Docker to set up environment for integration tests.

Re: [PR] Kafka Connect: Add config to route to tables using topic name [iceberg]

2024-12-29 Thread via GitHub
xiasongh commented on PR #11313: URL: https://github.com/apache/iceberg/pull/11313#issuecomment-2564847217 Hey @bryanck @mun1r0b0t! Sorry! I've been a bit busy and this PR got lost. I can finish up this PR, but I'd like to get some thoughts on my previous comment so I can make

Re: [I] throws "Caused by: org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at location:" [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on issue #10509: URL: https://github.com/apache/iceberg/issues/10509#issuecomment-2564877764 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] throws "Caused by: org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at location:" [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] closed issue #10509: throws "Caused by: org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at location:" URL: https://github.com/apache/iceberg/issues/10509 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [I] Describes how to set up a catalog that starts with the Rest Catalog(like Unity Catalog). [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] closed issue #10519: Describes how to set up a catalog that starts with the Rest Catalog(like Unity Catalog). URL: https://github.com/apache/iceberg/issues/10519 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Describes how to set up a catalog that starts with the Rest Catalog(like Unity Catalog). [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on issue #10519: URL: https://github.com/apache/iceberg/issues/10519#issuecomment-256488 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Overwrite with condition from spark failed [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on issue #10619: URL: https://github.com/apache/iceberg/issues/10619#issuecomment-2564877839 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Core,Format: Deprecate embedded manifests [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on PR #11586: URL: https://github.com/apache/iceberg/pull/11586#issuecomment-2564878805 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Handling NO Coordinator Scenario and Data Loss in the current Design [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on PR #11298: URL: https://github.com/apache/iceberg/pull/11298#issuecomment-2564878061 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core: Set missing table-default property in RESTSessionCatalog [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on PR #11646: URL: https://github.com/apache/iceberg/pull/11646#issuecomment-2564878961 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Handling NO Coordinator Scenario and Data Loss in the current Design [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] closed pull request #11298: Handling NO Coordinator Scenario and Data Loss in the current Design URL: https://github.com/apache/iceberg/pull/11298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Concurrency issues in `o.a.i.hive.MetastoreLock` [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on issue #10595: URL: https://github.com/apache/iceberg/issues/10595#issuecomment-2564877814 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Spark: 4.0 snapshot support [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] commented on PR #11583: URL: https://github.com/apache/iceberg/pull/11583#issuecomment-2564878514 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core,Format: Deprecate embedded manifests [iceberg]

2024-12-29 Thread via GitHub
github-actions[bot] closed pull request #11586: Core,Format: Deprecate embedded manifests URL: https://github.com/apache/iceberg/pull/11586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-29 Thread via GitHub
jiakai-li commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899180251 ## pyiceberg/io/pyarrow.py: ## @@ -362,6 +362,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste "region"

Re: [PR] docs: various spelling fixes [iceberg-python]

2024-12-29 Thread via GitHub
kevinjqliu commented on PR #1471: URL: https://github.com/apache/iceberg-python/pull/1471#issuecomment-2564996100 thank you! this tool is very helpful. Do you think its useful to include it as part of pre-commit hook? https://github.com/codespell-project/codespell?tab=readme-ov-file#pre

[I] Iceberg has data consistency issues when processing CDC deleted records [iceberg]

2024-12-29 Thread via GitHub
beyond-up opened a new issue, #11888: URL: https://github.com/apache/iceberg/issues/11888 ### Apache Iceberg version 1.5.2 ### Query engine Flink ### Please describe the bug 🐞 ![image](https://github.com/user-attachments/assets/836cca53-810c-47f1-b477-27839

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
liurenjie1024 commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899291231 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// F

Re: [I] i meet a question,time travel has some error to the query of timestamp [iceberg]

2024-12-29 Thread via GitHub
hashmapybx commented on issue #11889: URL: https://github.com/apache/iceberg/issues/11889#issuecomment-2565089773 > Thanks @hashmapybx for raising this question, unfortunately, the information that you provide is very little. At first glance, the SQL seems to be correct. What kind of versio

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899343049 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// For ord

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899344509 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// For ord

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899344509 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// For ord

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899341695 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// For ord

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-29 Thread via GitHub
ZENOTME commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1899342562 ## crates/iceberg/src/spec/schema.rs: ## @@ -1132,6 +1132,213 @@ impl ReassignFieldIds { } } +/// A post order schema visitor with partner. +/// +/// For ord