Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
nastra commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1778103857 ## api/src/main/java/org/apache/iceberg/util/WrapperSet.java: ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
nastra commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1778100059 ## api/src/main/java/org/apache/iceberg/util/DataFileSet.java: ## @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
nastra commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1778100745 ## api/src/main/java/org/apache/iceberg/util/DataFileSet.java: ## @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
nastra commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1778092264 ## api/src/main/java/org/apache/iceberg/util/DataFileSet.java: ## @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
nastra commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1778090237 ## api/src/main/java/org/apache/iceberg/util/WrapperSet.java: ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [I] Support Nessie catalog [iceberg-python]

2024-09-26 Thread via GitHub
cee-shubham commented on issue #19: URL: https://github.com/apache/iceberg-python/issues/19#issuecomment-2378455568 I want to create iceberg tables using pyiceberg and store it in minio store, so for this i have created docker containers for services named as: nessie, minio, dremio Earli

Re: [PR] Spec: Fix table of content generation [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on PR #11067: URL: https://github.com/apache/iceberg/pull/11067#issuecomment-2378429916 ping @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: Add DataFileSet / DeleteFileSet [iceberg]

2024-09-26 Thread via GitHub
aokolnychyi commented on code in PR #11195: URL: https://github.com/apache/iceberg/pull/11195#discussion_r1777999151 ## api/src/main/java/org/apache/iceberg/util/WrapperSet.java: ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Spark: Deprecate SparkAppenderFactory [iceberg]

2024-09-26 Thread via GitHub
aokolnychyi commented on PR #11076: URL: https://github.com/apache/iceberg/pull/11076#issuecomment-2378384646 Thanks, @ajantha-bhat! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Spark: Deprecate SparkAppenderFactory [iceberg]

2024-09-26 Thread via GitHub
aokolnychyi merged PR #11076: URL: https://github.com/apache/iceberg/pull/11076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [I] Inconsistent row count across versions [iceberg-python]

2024-09-26 Thread via GitHub
sungwy closed issue #1132: Inconsistent row count across versions URL: https://github.com/apache/iceberg-python/issues/1132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
a-agmon commented on PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#issuecomment-2378363853 Thanks again @manuzhang. @liurenjie1024 - please see this PR adding UT to this usage - creating a table provider from metadata file https://github.com/apache/iceberg-rust/pull/65

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
a-agmon commented on code in PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#discussion_r1777960263 ## crates/integrations/datafusion/src/table.rs: ## @@ -56,6 +56,13 @@ impl IcebergTableProvider { Ok(IcebergTableProvider { table, schema }) } + +

[PR] add test for table provider creation [iceberg-rust]

2024-09-26 Thread via GitHub
a-agmon opened a new pull request, #651: URL: https://github.com/apache/iceberg-rust/pull/651 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] [Minor][Test] Fix TestFastAppend.testAddManyFiles() [iceberg]

2024-09-26 Thread via GitHub
anuragmantri commented on PR #11218: URL: https://github.com/apache/iceberg/pull/11218#issuecomment-2378326817 Thanks for merging @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777916406 ## core/src/test/java/org/apache/iceberg/TestTables.java: ## @@ -93,6 +93,26 @@ public static TestTable create( return new TestTable(ops, name, reporter);

Re: [PR] feat: implement IcebergTableProviderFactory for datafusion [iceberg-rust]

2024-09-26 Thread via GitHub
yukkit commented on PR #600: URL: https://github.com/apache/iceberg-rust/pull/600#issuecomment-2378319330 remind @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
manuzhang commented on code in PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#discussion_r1777943490 ## crates/integrations/datafusion/src/table.rs: ## @@ -56,6 +56,13 @@ impl IcebergTableProvider { Ok(IcebergTableProvider { table, schema }) } + +

Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on issue #8932: URL: https://github.com/apache/iceberg/issues/8932#issuecomment-2378158489 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
a-agmon commented on PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#issuecomment-2378294205 Thanks @manuzhang , I will also add an integration test to cover this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
liurenjie1024 commented on PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#issuecomment-2378289659 > Not familiar with rust, but do we have tests for such changes? Good point, this is mostly an access modifier change, and other parts are already covered by test, so gener

Re: [I] [feature request] Support reading equality delete files [iceberg-python]

2024-09-26 Thread via GitHub
sungwy commented on issue #1210: URL: https://github.com/apache/iceberg-python/issues/1210#issuecomment-2378262870 This will be a fantastic addition to PyIceberg! Thank you for raising this issue @kevinjqliu and @Zyiqin-Miranda 🎉 -- This is an automated message from the Apache Git Servi

Re: [PR] Bug Fix: Position Deletes + row_filter yields less data when the DataFile is large [iceberg-python]

2024-09-26 Thread via GitHub
sungwy merged PR #1141: URL: https://github.com/apache/iceberg-python/pull/1141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
manuzhang commented on PR #650: URL: https://github.com/apache/iceberg-rust/pull/650#issuecomment-2378277977 Not familiar with rust, but do we have tests for such changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] discuss: re-export arrow types [iceberg-rust]

2024-09-26 Thread via GitHub
liurenjie1024 commented on issue #636: URL: https://github.com/apache/iceberg-rust/issues/636#issuecomment-2378260850 +1 for this, which helps to decipher in compatible types with same name. cc @Xuanwo What do you think? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2378257408 @aokolnychyi: This PR is ready. But as we discussed previously, this PR wraps the `PartitionStats` into a `Record` as the writers cannot work with Iceberg internal objects yet.

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777915925 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,17 +371,17 @@ public void testPartitionStatsWithSchemaEvolution() throws Except

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777914781 ## core/src/main/java/org/apache/iceberg/data/PartitionStatsRecord.java: ## @@ -0,0 +1,170 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777914781 ## core/src/main/java/org/apache/iceberg/data/PartitionStatsRecord.java: ## @@ -0,0 +1,170 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[PR] Config File Handling [iceberg-go]

2024-09-26 Thread via GitHub
alex-kar opened a new pull request, #156: URL: https://github.com/apache/iceberg-go/pull/156 Handle config file `.iceberg-go.yaml` in the home directory. Closes #62 @zeroshade I noticed that `--catalog` has different functionality compare to pyiceberg cli, where it defines cat

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777914012 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -87,13 +90,10 @@ private static PartitionMap collectStats( PartitionMap statsMap =

Re: [PR] Core: Use avro compression properties from table properties when writing manifests and manifest lists (API change) [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8617: URL: https://github.com/apache/iceberg/pull/8617#issuecomment-2378157592 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8673: URL: https://github.com/apache/iceberg/pull/8673#issuecomment-2378158030 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-09-26 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1777912510 ## core/src/main/java/org/apache/iceberg/PartitionStats.java: ## @@ -249,4 +250,45 @@ public void set(int pos, T value) { throw new UnsupportedOperatio

Re: [PR] fix: DayTransform result type override and docs [iceberg-python]

2024-09-26 Thread via GitHub
kevinjqliu commented on PR #1208: URL: https://github.com/apache/iceberg-python/pull/1208#issuecomment-2378222143 > Basically, Spark stores day transformed partition values incorrectly in the metadata Thats an interesting find... The core Iceberg library is using `DateType` as the Re

Re: [PR] fix: DayTransform result type override and docs [iceberg-python]

2024-09-26 Thread via GitHub
kevinjqliu commented on PR #1208: URL: https://github.com/apache/iceberg-python/pull/1208#issuecomment-2378238683 devlist https://lists.apache.org/thread/2gq7b54nvc9q6f1j08l9lnzgm5onkmx5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] feat (datafusion): making IcebergTableProvider public to be used without a catalog [iceberg-rust]

2024-09-26 Thread via GitHub
liurenjie1024 merged PR #650: URL: https://github.com/apache/iceberg-rust/pull/650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] AWS: Throw relevant exception at namespaceExists check usage in Glue Catalog [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8533: URL: https://github.com/apache/iceberg/pull/8533#issuecomment-2378157073 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] To fix the table configuration documentation for write.distrubition-mode [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8517: To fix the table configuration documentation for write.distrubition-mode URL: https://github.com/apache/iceberg/pull/8517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8673: Core: Mark `503: added_snapshot_id` as required URL: https://github.com/apache/iceberg/pull/8673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Core: Add AsyncFileIO [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8644: Core: Add AsyncFileIO URL: https://github.com/apache/iceberg/pull/8644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] multi-arg transform support [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed issue #8258: multi-arg transform support URL: https://github.com/apache/iceberg/issues/8258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Core: Refactor RestCatalogAdapter to use existing HttpMethod enum [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8543: Core: Refactor RestCatalogAdapter to use existing HttpMethod enum URL: https://github.com/apache/iceberg/pull/8543 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [Minor][Test] Fix TestFastAppend.testAddManyFiles() [iceberg]

2024-09-26 Thread via GitHub
amogh-jahagirdar merged PR #11218: URL: https://github.com/apache/iceberg/pull/11218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spec: Clarify the semantics of `identifier-field-ids` [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8623: URL: https://github.com/apache/iceberg/pull/8623#issuecomment-2378157661 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Flink: new sink base on the unified sink API - WIP [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8653: Flink: new sink base on the unified sink API - WIP URL: https://github.com/apache/iceberg/pull/8653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Use avro compression properties from table properties when writing manifests and manifest lists (API change) [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8617: Core: Use avro compression properties from table properties when writing manifests and manifest lists (API change) URL: https://github.com/apache/iceberg/pull/8617 -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Spec: Add section on `null_value_counts` [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8611: Spec: Add section on `null_value_counts` URL: https://github.com/apache/iceberg/pull/8611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Make FanoutWriter a package public class [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8581: Make FanoutWriter a package public class URL: https://github.com/apache/iceberg/pull/8581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [Minor][Test] Fix TestFastAppend.testAddManyFiles() [iceberg]

2024-09-26 Thread via GitHub
amogh-jahagirdar commented on PR #11218: URL: https://github.com/apache/iceberg/pull/11218#issuecomment-2378154237 I'll go ahead and merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] MERGE INTO number of affected rows [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed issue #8229: MERGE INTO number of affected rows URL: https://github.com/apache/iceberg/issues/8229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Core: Add AsyncFileIO [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8644: URL: https://github.com/apache/iceberg/pull/8644#issuecomment-2378157808 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Prevent dropping last column. [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8523: URL: https://github.com/apache/iceberg/pull/8523#issuecomment-2378157019 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] upgrade iceberg from 0.14.1 to 1.2.1, flink task error with InvalidClassException: org.apache.iceberg.BaseFileScanTask; local class incompatible: stream classdesc serialVersionUID = -410451952

2024-09-26 Thread via GitHub
github-actions[bot] commented on issue #8220: URL: https://github.com/apache/iceberg/issues/8220#issuecomment-2378156781 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Make FanoutWriter a package public class [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8581: URL: https://github.com/apache/iceberg/pull/8581#issuecomment-2378157324 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core: Widen exceptions ignored while deleting files in RollingFileWriter [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8597: URL: https://github.com/apache/iceberg/pull/8597#issuecomment-2378157477 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spec: Clarify the semantics of `identifier-field-ids` [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8623: Spec: Clarify the semantics of `identifier-field-ids` URL: https://github.com/apache/iceberg/pull/8623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Flink: new sink base on the unified sink API - WIP [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8653: URL: https://github.com/apache/iceberg/pull/8653#issuecomment-2378157962 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core: Widen exceptions ignored while deleting files in RollingFileWriter [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8597: Core: Widen exceptions ignored while deleting files in RollingFileWriter URL: https://github.com/apache/iceberg/pull/8597 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] RollingFileWriter Throws Exceptions if it Does Not Have Delete Permissions [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed issue #8253: RollingFileWriter Throws Exceptions if it Does Not Have Delete Permissions URL: https://github.com/apache/iceberg/issues/8253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spec: Add section on `null_value_counts` [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8611: URL: https://github.com/apache/iceberg/pull/8611#issuecomment-2378157544 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Prevent dropping last column. [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8523: Prevent dropping last column. URL: https://github.com/apache/iceberg/pull/8523 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] AWS: Throw relevant exception at namespaceExists check usage in Glue Catalog [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] closed pull request #8533: AWS: Throw relevant exception at namespaceExists check usage in Glue Catalog URL: https://github.com/apache/iceberg/pull/8533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] To fix the table configuration documentation for write.distrubition-mode [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8517: URL: https://github.com/apache/iceberg/pull/8517#issuecomment-2378156981 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] RollingFileWriter Throws Exceptions if it Does Not Have Delete Permissions [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on issue #8253: URL: https://github.com/apache/iceberg/issues/8253#issuecomment-2378156836 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] upgrade iceberg from 0.14.1 to 1.2.1, flink task error with InvalidClassException: org.apache.iceberg.BaseFileScanTask; local class incompatible: stream classdesc serialVersionUID = -410451952

2024-09-26 Thread via GitHub
github-actions[bot] closed issue #8220: upgrade iceberg from 0.14.1 to 1.2.1, flink task error with InvalidClassException: org.apache.iceberg.BaseFileScanTask; local class incompatible: stream classdesc serialVersionUID = -4104519524386879586, local class serialVersionUID = -44073174547117222

Re: [PR] Core: Refactor RestCatalogAdapter to use existing HttpMethod enum [iceberg]

2024-09-26 Thread via GitHub
github-actions[bot] commented on PR #8543: URL: https://github.com/apache/iceberg/pull/8543#issuecomment-2378157109 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777853976 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] AWS: Set better defaults for S3 retry behaviour [iceberg]

2024-09-26 Thread via GitHub
amogh-jahagirdar commented on code in PR #11052: URL: https://github.com/apache/iceberg/pull/11052#discussion_r1777830154 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -824,6 +889,65 @@ public void applyEndpointConfigurations(T builder) { }

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-09-26 Thread via GitHub
rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r181725 ## format/spec.md: ## @@ -298,16 +298,137 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] AWS: Set better defaults for S3 retry behaviour [iceberg]

2024-09-26 Thread via GitHub
danielcweeks commented on code in PR #11052: URL: https://github.com/apache/iceberg/pull/11052#discussion_r153530 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -824,6 +889,65 @@ public void applyEndpointConfigurations(T builder) { }

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r165276 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java: ## @@ -0,0 +1,326 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r163032 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java: ## @@ -0,0 +1,326 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r162669 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java: ## @@ -0,0 +1,326 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-09-26 Thread via GitHub
rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r161549 ## format/spec.md: ## @@ -298,16 +298,143 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r144352 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-09-26 Thread via GitHub
rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r157402 ## format/spec.md: ## @@ -298,16 +298,143 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [I] DOCS: Missing Docstrings in public API [iceberg-python]

2024-09-26 Thread via GitHub
sungwy commented on issue #1056: URL: https://github.com/apache/iceberg-python/issues/1056#issuecomment-2377221051 Hi @askalik - yes, that'll be amazing! If you leave a comment on #1191 I can get that assigned to you -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r144788 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
rodmeneses commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r142164 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on PR #11144: URL: https://github.com/apache/iceberg/pull/11144#issuecomment-2377941746 @rodmeneses: If you have time, I would like to ask you to review the PR. Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on PR #11144: URL: https://github.com/apache/iceberg/pull/11144#issuecomment-2377940857 @netvl: I'm struggling to find the relevant comment (there are too many of them already), but I would like to ask you to check if the proposed `MaintenanceTaskBuilder` would serve your ne

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r132933 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r132933 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r131546 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r131032 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestExpireSnapshots.java: ## @@ -0,0 +1,235 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r130668 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] AWS: Set better defaults for S3 retry behaviour [iceberg]

2024-09-26 Thread via GitHub
danielcweeks commented on code in PR #11052: URL: https://github.com/apache/iceberg/pull/11052#discussion_r1777363011 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -393,6 +403,21 @@ public class S3FileIOProperties implements Serializable { */

Re: [PR] fix: DayTransform result type override and docs [iceberg-python]

2024-09-26 Thread via GitHub
kevinzwang commented on PR #1208: URL: https://github.com/apache/iceberg-python/pull/1208#issuecomment-2377884626 I played around with Spark and inspected the generated metadata. I believe the partition value is actually stored in the metadata as date type for day transformed partitions.

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-09-26 Thread via GitHub
rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1777691887 ## format/spec.md: ## @@ -454,29 +581,29 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777436962 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] PR #1169 [iceberg-python]

2024-09-26 Thread via GitHub
kevinjqliu commented on code in PR #1206: URL: https://github.com/apache/iceberg-python/pull/1206#discussion_r1777321794 ## pyiceberg/io/pyarrow.py: ## @@ -1068,20 +1068,13 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType: return StringType()

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777677656 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777664609 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java: ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777629218 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777664609 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java: ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Soft

Re: [I] Support relative paths in Table Metadata [iceberg]

2024-09-26 Thread via GitHub
devinrsmith commented on issue #1617: URL: https://github.com/apache/iceberg/issues/1617#issuecomment-2377813382 I'm primarily interested this as a means to enable unit testing; pyicberg seems to be able to read and write relative paths of the form: * `my-warehouse/...` * `file:my-

Re: [I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-09-26 Thread via GitHub
BTheunissen commented on issue #1104: URL: https://github.com/apache/iceberg-python/issues/1104#issuecomment-2377397379 @cshenrik Sorry about the lateness, I actually did a small internal fork of the library and added the following logic to: ``` def _initialize_fs(self, scheme: str, n

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777615921 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java: ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-26 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1777650877 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

  1   2   3   >