Re: [PR] Build: Bump org.xerial.snappy:snappy-java from 1.1.10.6 to 1.1.10.7 [iceberg]

2024-09-18 Thread via GitHub
nastra commented on PR #11140: URL: https://github.com/apache/iceberg/pull/11140#issuecomment-2360108312 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Table has more than one bucket keys, but "show create table xxx" only displays one [iceberg]

2024-09-18 Thread via GitHub
madeirak commented on issue #11090: URL: https://github.com/apache/iceberg/issues/11090#issuecomment-2360131236 > The table has two partition keys from two [partition transforms](https://iceberg.apache.org/spec/#partition-transforms), one of which is bucket. ![image](https://github.

Re: [PR] Build: Bump org.apache.datasketches:datasketches-java from 6.0.0 to 6.1.0 [iceberg]

2024-09-18 Thread via GitHub
nastra commented on PR #11137: URL: https://github.com/apache/iceberg/pull/11137#issuecomment-2360122519 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Build: Add .java-version to gitignore. [iceberg]

2024-09-18 Thread via GitHub
szehon-ho commented on PR #11167: URL: https://github.com/apache/iceberg/pull/11167#issuecomment-2360123374 Yes good solution to allow having .java_version, looks good to me too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Table has more than one bucket keys, but "show create table xxx" only displays one [iceberg]

2024-09-18 Thread via GitHub
manuzhang commented on issue #11090: URL: https://github.com/apache/iceberg/issues/11090#issuecomment-2360118603 The table has two partition keys from two partition transforms, one of which is bucket. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] execute sql "select partition from dbxx.tbxx.partitons" on table which has never had partition field get error❌ [iceberg]

2024-09-18 Thread via GitHub
manuzhang commented on issue #11105: URL: https://github.com/apache/iceberg/issues/11105#issuecomment-2360111261 As documented at https://iceberg.apache.org/docs/nightly/spark-queries/#partitions > For unpartitioned tables, the partitions table will not contain the partition and spec

Re: [PR] Build: Add .java-version to gitignore. [iceberg]

2024-09-18 Thread via GitHub
nastra merged PR #11167: URL: https://github.com/apache/iceberg/pull/11167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add explicit JSON parser for LoadTableResponse [iceberg]

2024-09-18 Thread via GitHub
nastra merged PR #11148: URL: https://github.com/apache/iceberg/pull/11148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add explicit JSON parser for LoadTableResponse [iceberg]

2024-09-18 Thread via GitHub
nastra commented on PR #11148: URL: https://github.com/apache/iceberg/pull/11148#issuecomment-2360031810 thanks @amogh-jahagirdar for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Flink-1.19: Fix the file offset mismatch when Flink reader first seek… [iceberg]

2024-09-18 Thread via GitHub
zhongyujiang commented on PR #10567: URL: https://github.com/apache/iceberg/pull/10567#issuecomment-2359964996 Hi @pvary @stevenzwu, gentle ping. Can you please help review this when you have time? Although this PR does not provide an e2e reproduction case, I believe the bug in `DataIterato

[PR] Build: Add .java-version to gitignore. [iceberg]

2024-09-18 Thread via GitHub
zhongyujiang opened a new pull request, #11167: URL: https://github.com/apache/iceberg/pull/11167 Add the .java-version file to .gitignore, so users can customize their preferred JDK version locally without affecting the project. This can help users who manage multiple JDK versions using Je

Re: [PR] Build: Bump Spark 3.5 to 3.5.3 [iceberg]

2024-09-18 Thread via GitHub
manuzhang commented on PR #11160: URL: https://github.com/apache/iceberg/pull/11160#issuecomment-2359915308 Tests failure look to be caused by https://github.com/apache/spark/pull/47724. I'm checking with Spark community. -- This is an automated message from the Apache Git Service. To res

Re: [PR] Core: Fix UnicodeUtil#truncateStringMax returns malformed string. [iceberg]

2024-09-18 Thread via GitHub
zhongyujiang commented on PR #11161: URL: https://github.com/apache/iceberg/pull/11161#issuecomment-2359897318 @amogh-jahagirdar @nastra can you please help review this? thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Encryption integration and test [iceberg]

2024-09-18 Thread via GitHub
hsiang-c commented on code in PR #5544: URL: https://github.com/apache/iceberg/pull/5544#discussion_r1766090302 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -110,6 +115,15 @@ public void initialize(String inputName, Map properties) {

Re: [I] Config File Handling [iceberg-go]

2024-09-18 Thread via GitHub
alex-kar commented on issue #62: URL: https://github.com/apache/iceberg-go/issues/62#issuecomment-2359890609 @zeroshade What's the initial structure for configuration file would look like? Something similar to `pyiceberg`'s config: ``` catalog: default: catalog: rest

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
wgtmac commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766076050 ## format/spec.md: ## @@ -454,28 +466,28 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
Kontinuation commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766071261 ## format/spec.md: ## @@ -454,28 +466,28 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
wgtmac commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766069450 ## format/spec.md: ## @@ -454,28 +466,28 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766044077 ## format/spec.md: ## @@ -454,28 +466,28 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the fol

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766037796 ## format/spec.md: ## @@ -200,12 +200,16 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
wgtmac commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1766027640 ## format/spec.md: ## @@ -454,28 +466,28 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

2024-09-18 Thread via GitHub
amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1765979097 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -153,6 +154,12 @@ void caseSensitive(boolean newCaseSensitive) { void delete(F

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

2024-09-18 Thread via GitHub
amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1765980819 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -308,11 +316,15 @@ private ManifestFile filterManifest(Schema tableSchema, Manife

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

2024-09-18 Thread via GitHub
amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1765979097 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -153,6 +154,12 @@ void caseSensitive(boolean newCaseSensitive) { void delete(F

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

2024-09-18 Thread via GitHub
amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1765979097 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -153,6 +154,12 @@ void caseSensitive(boolean newCaseSensitive) { void delete(F

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

2024-09-18 Thread via GitHub
amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1765979097 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -153,6 +154,12 @@ void caseSensitive(boolean newCaseSensitive) { void delete(F

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-18 Thread via GitHub
wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765955792 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-18 Thread via GitHub
wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765955792 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] [spark 3.4] add remove_empty_files procedure [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8051: [spark 3.4] add remove_empty_files procedure URL: https://github.com/apache/iceberg/pull/8051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-09-18 Thread via GitHub
stevenzwu commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1765512677 ## format/spec.md: ## @@ -298,16 +298,137 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] [spark 3.4] add remove_empty_files procedure [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8051: URL: https://github.com/apache/iceberg/pull/8051#issuecomment-2359659661 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Docs: Add documentation for Rate limiting in Spark Structured Streaming [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8044: URL: https://github.com/apache/iceberg/pull/8044#issuecomment-2359659622 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] AWS: Add retry on UncheckedIOException and max retries for S3FileIO [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8043: AWS: Add retry on UncheckedIOException and max retries for S3FileIO URL: https://github.com/apache/iceberg/pull/8043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] AWS: Add retry on UncheckedIOException and max retries for S3FileIO [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8043: URL: https://github.com/apache/iceberg/pull/8043#issuecomment-2359659606 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark 3.3: Adding Rebalance operator solving for small files problem [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8042: URL: https://github.com/apache/iceberg/pull/8042#issuecomment-2359659583 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark 3.3: Adding Rebalance operator solving for small files problem [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8042: Spark 3.3: Adding Rebalance operator solving for small files problem URL: https://github.com/apache/iceberg/pull/8042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Data : Standardize the error message in TableMigrationUtil [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8041: Data : Standardize the error message in TableMigrationUtil URL: https://github.com/apache/iceberg/pull/8041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [spark 3.4] skip empty file during table migration, table snapshotting or adding files [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8040: URL: https://github.com/apache/iceberg/pull/8040#issuecomment-2359659530 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] [spark 3.4] skip empty file during table migration, table snapshotting or adding files [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8040: [spark 3.4] skip empty file during table migration, table snapshotting or adding files URL: https://github.com/apache/iceberg/pull/8040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Build: Apply spotless for scala code [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8023: Build: Apply spotless for scala code URL: https://github.com/apache/iceberg/pull/8023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Core: Avoid concurrent commits causing commit failures [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #8001: Core: Avoid concurrent commits causing commit failures URL: https://github.com/apache/iceberg/pull/8001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] FlinkSQL Upsert did'nt support timestamp column as a primary key [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed issue #7707: FlinkSQL Upsert did'nt support timestamp column as a primary key URL: https://github.com/apache/iceberg/issues/7707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Duplicate records with MERGE command [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on issue #7005: URL: https://github.com/apache/iceberg/issues/7005#issuecomment-2359659310 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Duplicate records with MERGE command [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed issue #7005: Duplicate records with MERGE command URL: https://github.com/apache/iceberg/issues/7005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Delete/Update fails for tables with more than 1000 columns [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on issue #6368: URL: https://github.com/apache/iceberg/issues/6368#issuecomment-2359659275 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Encryption integration and test [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #5544: URL: https://github.com/apache/iceberg/pull/5544#issuecomment-2359659257 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Add FileIO implementation for Azure Blob Storage [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #4465: URL: https://github.com/apache/iceberg/pull/4465#issuecomment-2359659238 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Data : Standardize the error message in TableMigrationUtil [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8041: URL: https://github.com/apache/iceberg/pull/8041#issuecomment-2359659553 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core: Avoid concurrent commits causing commit failures [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #8001: URL: https://github.com/apache/iceberg/pull/8001#issuecomment-2359659461 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Remove `com.palantir.baseline-exact-dependencies` plugin [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #7968: Remove `com.palantir.baseline-exact-dependencies` plugin URL: https://github.com/apache/iceberg/pull/7968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Remove `com.palantir.baseline-exact-dependencies` plugin [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on PR #7968: URL: https://github.com/apache/iceberg/pull/7968#issuecomment-2359659417 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] FlinkSQL Upsert did'nt support timestamp column as a primary key [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on issue #7707: URL: https://github.com/apache/iceberg/issues/7707#issuecomment-2359659356 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Support Rewrite Datafiles into a custom Partition Spec [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] commented on issue #7557: URL: https://github.com/apache/iceberg/issues/7557#issuecomment-2359659331 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Support Rewrite Datafiles into a custom Partition Spec [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed issue #7557: Support Rewrite Datafiles into a custom Partition Spec URL: https://github.com/apache/iceberg/issues/7557 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add FileIO implementation for Azure Blob Storage [iceberg]

2024-09-18 Thread via GitHub
github-actions[bot] closed pull request #4465: Add FileIO implementation for Azure Blob Storage URL: https://github.com/apache/iceberg/pull/4465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Support Snapshot Expiration Operation [iceberg-python]

2024-09-18 Thread via GitHub
ndrluis commented on issue #516: URL: https://github.com/apache/iceberg-python/issues/516#issuecomment-2359658336 @kevinjqliu I believe that I now understand the differences in how we perform TableMetadata updates in Python versus how it's done in Java. I think that the set of classes descr

Re: [PR] API, Core: Enable dropping rewritten delete files in RowDelta [iceberg]

2024-09-18 Thread via GitHub
aokolnychyi commented on code in PR #11166: URL: https://github.com/apache/iceberg/pull/11166#discussion_r1765893787 ## api/src/main/java/org/apache/iceberg/RowDelta.java: ## @@ -46,6 +46,17 @@ public interface RowDelta extends SnapshotUpdate { */ RowDelta addDeletes(Dele

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
dmitrykoval commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1765870308 ## format/spec.md: ## @@ -200,12 +200,16 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
dmitrykoval commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1765870308 ## format/spec.md: ## @@ -200,12 +200,16 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [I] Does main branch reference reset requiring a clean up of snapshot logs [iceberg]

2024-09-18 Thread via GitHub
haizhou-zhao commented on issue #11109: URL: https://github.com/apache/iceberg/issues/11109#issuecomment-2359610128 Down to code level detail (root cause of why REST differs from Hadoop/Hive), these two methods, seemingly doing similar things when resetting main branch, but one would clear

Re: [PR] Preserve last-updated-ms field in TableMetadata rebuild [iceberg]

2024-09-18 Thread via GitHub
haizhou-zhao closed pull request #11107: Preserve last-updated-ms field in TableMetadata rebuild URL: https://github.com/apache/iceberg/pull/11107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Bump mkdocs-material from 9.5.34 to 9.5.35 [iceberg-python]

2024-09-18 Thread via GitHub
dependabot[bot] opened a new pull request, #1184: URL: https://github.com/apache/iceberg-python/pull/1184 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.34 to 9.5.35. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

Re: [PR] OpenAPI: Add query param to control namespace separator [iceberg]

2024-09-18 Thread via GitHub
sungwy commented on code in PR #10904: URL: https://github.com/apache/iceberg/pull/10904#discussion_r1765739027 ## open-api/rest-catalog-open-api.yaml: ## Review Comment: Hi @nastra - the proposal looks good to me. I just did a quick scan through the endpoints to see if we

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765733792 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Docs: Clarified Partition Transform [iceberg]

2024-09-18 Thread via GitHub
jasonmfehr commented on PR #8337: URL: https://github.com/apache/iceberg/pull/8337#issuecomment-2359417376 @nastra -- done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765719946 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765720264 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765703804 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765701035 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765707969 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765700118 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765697595 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] FIX: Exception Handling in AWS Glue renameTable Method [iceberg]

2024-09-18 Thread via GitHub
jishangarg commented on PR #11165: URL: https://github.com/apache/iceberg/pull/11165#issuecomment-2359327812 Can somebody please review my PR ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Ability to pickle the `Catalog` [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on issue #514: URL: https://github.com/apache/iceberg-python/issues/514#issuecomment-2359301397 More context: https://github.com/ray-project/ray/pull/42235#discussion_r1520929199 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Ability to pickle the `Catalog` [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on issue #514: URL: https://github.com/apache/iceberg-python/issues/514#issuecomment-235928 @dev-goyal I see that you've merged [ray-project/ray#46889](https://github.com/ray-project/ray/pull/46889) curious if you think pickling the `Catalog` or `Table` is still val

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-18 Thread via GitHub
rdblue commented on PR #11001: URL: https://github.com/apache/iceberg/pull/11001#issuecomment-2359292156 Thanks for doing the research and getting this in, @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Revert "Cache Manifest files" [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on PR #1167: URL: https://github.com/apache/iceberg-python/pull/1167#issuecomment-2359286092 @sungwy here you go https://gist.github.com/kevinjqliu/a0e8e2199bd8064757eb2b40409e0794 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] Table rename in Glue Catalog throws Incorrect `AlreadyExistsException` [iceberg]

2024-09-18 Thread via GitHub
jishangarg commented on issue #11155: URL: https://github.com/apache/iceberg/issues/11155#issuecomment-2359263800 Hi Team, Can somebody please review my PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] Inconsistent id definition on Flink resolvedSchema conversion to iceberg schema [iceberg]

2024-09-18 Thread via GitHub
pvary commented on issue #11128: URL: https://github.com/apache/iceberg/issues/11128#issuecomment-2359257094 I'm facing a similar challenge. See: https://lists.apache.org/thread/vyw595d0747p33qg886b1o82mcw40523 The visitors could be used to traverse the schema, but you need to match

[PR] FIX: Exception Handling in AWS Glue renameTable Method [iceberg]

2024-09-18 Thread via GitHub
jishangarg opened a new pull request, #11165: URL: https://github.com/apache/iceberg/pull/11165 This PR addresses an issue in the AWS Glue renameTable method where an incorrect exception is thrown when a table with the new name already exists. According to the Iceberg Catalog interface, the

Re: [PR] Manifest list encryption [iceberg]

2024-09-18 Thread via GitHub
rdblue commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1765596130 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -160,4 +160,10 @@ private CatalogProperties() {} public static final String ENCRYPTION_KMS_TY

Re: [I] Incorrect schema used when using time-travel [iceberg]

2024-09-18 Thread via GitHub
jishangarg commented on issue #11162: URL: https://github.com/apache/iceberg/issues/11162#issuecomment-2359234009 Hi @fides-bot, can I know which version of Spark you are using? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Feature/otf 1500 column comparisons 1521 [iceberg]

2024-09-18 Thread via GitHub
jenbaldwin opened a new pull request, #11164: URL: https://github.com/apache/iceberg/pull/11164 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] List all AWS S3 properties in the docs [iceberg]

2024-09-18 Thread via GitHub
jishangarg commented on issue #10674: URL: https://github.com/apache/iceberg/issues/10674#issuecomment-2359206029 Hi, I would like to work on it, please assign me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765541651 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765542690 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestAsyncDeleteFiles.java: ## @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software

Re: [PR] Docs: Update MRAP endpoint and add more docs [iceberg]

2024-09-18 Thread via GitHub
eder001 commented on PR #11159: URL: https://github.com/apache/iceberg/pull/11159#issuecomment-2359168657 @edgarRd @nastra @amogh-jahagirdar @Fokko can one of you please evaluate those? -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765532417 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765524254 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765524051 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765523637 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765522216 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-18 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1765522579 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java: ## @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software F

Re: [PR] OpenAPI: Add planning-mode to loadTable response [iceberg]

2024-09-18 Thread via GitHub
jackye1995 commented on code in PR #11156: URL: https://github.com/apache/iceberg/pull/11156#discussion_r1765496741 ## open-api/rest-catalog-open-api.yaml: ## @@ -3240,6 +3253,11 @@ components: type: string enum: ["completed", "submitted", "cancelled", "failed"]

Re: [PR] OpenAPI: Add planning-mode to loadTable response [iceberg]

2024-09-18 Thread via GitHub
jackye1995 commented on code in PR #11156: URL: https://github.com/apache/iceberg/pull/11156#discussion_r1765495993 ## open-api/rest-catalog-open-api.yaml: ## @@ -3138,6 +3149,8 @@ components: description: May be null if the table is staged as part of a transaction

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-18 Thread via GitHub
dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765491165 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existi

[I] Make REST Catalog Namespace Separator Configurable [iceberg-python]

2024-09-18 Thread via GitHub
sungwy opened a new issue, #1183: URL: https://github.com/apache/iceberg-python/issues/1183 ### Feature Request / Improvement Similar to the proposed changes in Java/ REST Catalog Open API Spec, we'd like to make the REST Catalog Namespace Separator Configurable address issued in spe

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2359019036 Gotcha, thanks! I think the CI runs in ubuntu, so it should mirror your own setup -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
szehon-ho commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1765411351 ## format/spec.md: ## @@ -200,12 +200,16 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [PR] Spec: Support geo type [iceberg]

2024-09-18 Thread via GitHub
szehon-ho commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1765411351 ## format/spec.md: ## @@ -200,12 +200,16 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

  1   2   >