Re: [PR] Manifest list encryption [iceberg]

2024-09-17 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1762446655 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -160,4 +160,10 @@ private CatalogProperties() {} public static final String ENCRYPTION_K

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-09-17 Thread via GitHub
nqvuong1998 commented on PR #9852: URL: https://github.com/apache/iceberg/pull/9852#issuecomment-2354712766 Hi @nastra @danielcweeks , Is this pull request scheduled for inclusion in the v1.7 release? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-09-17 Thread via GitHub
nastra commented on PR #9852: URL: https://github.com/apache/iceberg/pull/9852#issuecomment-2354729443 @nqvuong1998 yes that will be shipped with 1.7.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762551328 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762557022 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762578527 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762592329 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762593581 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762594886 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

[PR] Core: Add explicit parser for LoadTableResponse [iceberg]

2024-09-17 Thread via GitHub
nastra opened a new pull request, #11148: URL: https://github.com/apache/iceberg/pull/11148 This introduces an explicit JSON parser for `LoadTableResponse` as a preparation step for standardizing credentials as proposed in https://github.com/apache/iceberg/pull/10722. Currently, `Loa

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762616215 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762617704 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762618924 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762620597 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762620981 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762646687 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762649970 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762654347 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762656115 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762671806 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762673446 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java: ## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762674501 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java: ## @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762677420 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java: ## @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762701329 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1762731921 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Avoid metaspace memory leak by not registering ShutdownHook for ExecutorService in Flink [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11073: URL: https://github.com/apache/iceberg/pull/11073#discussion_r1762769631 ## core/src/main/java/org/apache/iceberg/util/ThreadPools.java: ## @@ -86,9 +86,18 @@ public static ExecutorService newWorkerPool(String namePrefix) { } public

[I] rewrite_data_files procedure is compatible ranger auth check [iceberg]

2024-09-17 Thread via GitHub
ludlows opened a new issue, #11149: URL: https://github.com/apache/iceberg/issues/11149 ### Apache Iceberg version 1.6.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 in the code of `rewrite_data_files` procedure: ```java Datase

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1762795906 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [I] Iceberg to configure AWS S3 configuration with the Hadoop and Hive4 setup is hanging without giving ant error [iceberg]

2024-09-17 Thread via GitHub
pvary commented on issue #11145: URL: https://github.com/apache/iceberg/issues/11145#issuecomment-2355021232 @AwasthiSomesh: The issue name suggests that this problem happens with Hive4. That is why I suggested that the **Apache Hive team could help you** better. The Hive 4 integration is m

Re: [I] Mixed usage of snapshotCreationTs, metadataCommitTs & tableAccessTs when using REST Catalog [iceberg]

2024-09-17 Thread via GitHub
nastra commented on issue #11103: URL: https://github.com/apache/iceberg/issues/11103#issuecomment-2355073606 I've opened https://github.com/apache/iceberg/pull/11151 to fix this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1762895735 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -578,6 +578,10 @@ public TableMetadata removeSnapshotsIf(Predicate removeIf) { return new Buil

Re: [I] Iceberg to configure AWS S3 configuration with the Hadoop and Hive4 setup is hanging without giving ant error [iceberg]

2024-09-17 Thread via GitHub
AwasthiSomesh commented on issue #11145: URL: https://github.com/apache/iceberg/issues/11145#issuecomment-2355122259 looks like hive issue discussion is not available through git-hub anyone knows how to reach out hive4 team via GitHub -- This is an automated message from the Apache Git

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1762895735 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -578,6 +578,10 @@ public TableMetadata removeSnapshotsIf(Predicate removeIf) { return new Buil

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1763067211 ## core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePath.java: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Propagate catalog-level properties to iceberg-views. [iceberg]

2024-09-17 Thread via GitHub
nk1506 closed pull request #11025: Core: Propagate catalog-level properties to iceberg-views. URL: https://github.com/apache/iceberg/pull/11025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Core: Propagate catalog-level properties to iceberg-views. [iceberg]

2024-09-17 Thread via GitHub
nk1506 commented on PR #11025: URL: https://github.com/apache/iceberg/pull/11025#issuecomment-2355701201 Closing this, I will push this on the ongoing pr https://github.com/apache/iceberg/pull/11064 -- This is an automated message from the Apache Git Service. To respond to the message, pl

[PR] Feature/otf 1500 column comparison [iceberg]

2024-09-17 Thread via GitHub
jenbaldwin opened a new pull request, #11152: URL: https://github.com/apache/iceberg/pull/11152 Draft PR for early feedback. (btw otf-1500 is internal jira number) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] REST: Handle Requests with Page Sizes Exceeding Available Number of Namespaces /Tables/Views [iceberg]

2024-09-17 Thread via GitHub
rcjverhoef commented on code in PR #11143: URL: https://github.com/apache/iceberg/pull/11143#discussion_r1763242975 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2347,7 +2347,7 @@ public void testPaginationForListNamespaces() { RESTCatalog catal

Re: [PR] REST: Handle Requests with Page Sizes Exceeding Available Number of Namespaces /Tables/Views [iceberg]

2024-09-17 Thread via GitHub
rcjverhoef commented on code in PR #11143: URL: https://github.com/apache/iceberg/pull/11143#discussion_r1763241863 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2409,7 +2409,7 @@ public void testPaginationForListTables() { RESTCatalog catalog =

Re: [PR] REST: Handle Requests with Page Sizes Exceeding Available Number of Namespaces /Tables/Views [iceberg]

2024-09-17 Thread via GitHub
rcjverhoef commented on code in PR #11143: URL: https://github.com/apache/iceberg/pull/11143#discussion_r1763241863 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2409,7 +2409,7 @@ public void testPaginationForListTables() { RESTCatalog catalog =

Re: [PR] REST: Handle Requests with Page Sizes Exceeding Available Number of Namespaces /Tables/Views [iceberg]

2024-09-17 Thread via GitHub
rcjverhoef commented on code in PR #11143: URL: https://github.com/apache/iceberg/pull/11143#discussion_r1763241863 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2409,7 +2409,7 @@ public void testPaginationForListTables() { RESTCatalog catalog =

Re: [I] deletion & purge improvements for undelete feature in REST catalog [iceberg]

2024-09-17 Thread via GitHub
twuebi commented on issue #11023: URL: https://github.com/apache/iceberg/issues/11023#issuecomment-2355892509 https://github.com/twuebi/iceberg/commit/90805de3b415a2dae6d9d8b952966d32e2db7ffe adds the property, after reading the code a bit more, I was left wondering why the SparkCatalog is

Re: [I] Iceberg to configure AWS S3 configuration with the Hadoop and Hive4 setup is hanging without giving ant error [iceberg]

2024-09-17 Thread via GitHub
AwasthiSomesh commented on issue #11145: URL: https://github.com/apache/iceberg/issues/11145#issuecomment-2357518430 @pvary Thanks a lot for your quick response . I have 2 below question could you please help me with your comments. **Q1**. As mentioned in iceberg official docum

Re: [I] Mixed usage of snapshotCreationTs, metadataCommitTs & tableAccessTs when using REST Catalog [iceberg]

2024-09-17 Thread via GitHub
nastra commented on issue #11103: URL: https://github.com/apache/iceberg/issues/11103#issuecomment-2357550633 fixed by https://github.com/apache/iceberg/pull/11151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Mixed usage of snapshotCreationTs, metadataCommitTs & tableAccessTs when using REST Catalog [iceberg]

2024-09-17 Thread via GitHub
nastra closed issue #11103: Mixed usage of snapshotCreationTs, metadataCommitTs & tableAccessTs when using REST Catalog URL: https://github.com/apache/iceberg/issues/11103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
nastra merged PR #11151: URL: https://github.com/apache/iceberg/pull/11151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add explicit JSON parser for LoadTableResponse [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #11148: URL: https://github.com/apache/iceberg/pull/11148#discussion_r1764428159 ## core/src/main/java/org/apache/iceberg/rest/responses/LoadTableResponse.java: ## @@ -61,7 +62,12 @@ public String metadataLocation() { } public TableMetadata

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1764467923 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -262,6 +264,120 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1764468077 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -262,6 +264,120 @@ public void testReadColumnFilter2() throws Exception {

Re: [I] Delete Files in Table Scans [iceberg-rust]

2024-09-17 Thread via GitHub
sdd commented on issue #630: URL: https://github.com/apache/iceberg-rust/issues/630#issuecomment-2357605851 I'm happy to add the partitioning result to the task. This is useful to the executor node when deciding how to distribute tasks, as it enables the use of a few different strategies, t

Re: [PR] Kafka Connect: separate CI workflow [iceberg]

2024-09-17 Thread via GitHub
nastra merged PR #11075: URL: https://github.com/apache/iceberg/pull/11075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2024-09-17 Thread via GitHub
hsiang-c commented on code in PR #9335: URL: https://github.com/apache/iceberg/pull/9335#discussion_r1764382306 ## .palantir/revapi.yml: ## @@ -1136,6 +1136,78 @@ acceptedBreaks: new: "method org.apache.iceberg.BaseMetastoreOperations.CommitStatus org.apache.iceberg.Base

[PR] OpenAPI: Add planning-mode to loadTable response [iceberg]

2024-09-17 Thread via GitHub
rahil-c opened a new pull request, #11156: URL: https://github.com/apache/iceberg/pull/11156 Recently in the iceberg community we landed a new set of scan planning apis within the rest spec https://github.com/apache/iceberg/pull/9695. The following spec change in this pr aims to prov

Re: [I] Create table format version constants [iceberg-python]

2024-09-17 Thread via GitHub
tanmayrauth commented on issue #851: URL: https://github.com/apache/iceberg-python/issues/851#issuecomment-2357506218 @kevinjqliu I found this TableVersion [declaration already present](https://github.com/apache/iceberg-python/blob/de47590c6ac4f507cb2337c20504a62c484339f9/pyiceberg/typedef.p

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-17 Thread via GitHub
sungwy commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2356057167 Hi @JE-Chen thank you for putting together this PR! It looks like these were missed in the initial refactoring. Running the CI now. -- This is an automated message from the Apache

Re: [I] Minimum required pyarrow version [iceberg-python]

2024-09-17 Thread via GitHub
sungwy commented on issue #1174: URL: https://github.com/apache/iceberg-python/issues/1174#issuecomment-2356063681 Hi @gli-chris-hao thank you for reporting this issue! @ndrluis has already made a contribution to [fix this in main](https://github.com/apache/iceberg-python/pull/1090) --

Re: [PR] Add flag to allow diabling creation of catalog tables [iceberg-python]

2024-09-17 Thread via GitHub
isc-patrick commented on code in PR #1155: URL: https://github.com/apache/iceberg-python/pull/1155#discussion_r1763399699 ## tests/catalog/test_sql.py: ## @@ -225,6 +237,93 @@ def test_creation_from_impl(catalog_name: str, warehouse: Path) -> None: ) +def confirm_no_ta

Re: [PR] Add flag to allow diabling creation of catalog tables [iceberg-python]

2024-09-17 Thread via GitHub
isc-patrick commented on code in PR #1155: URL: https://github.com/apache/iceberg-python/pull/1155#discussion_r1763400342 ## tests/catalog/test_sql.py: ## @@ -225,6 +237,93 @@ def test_creation_from_impl(catalog_name: str, warehouse: Path) -> None: ) +def confirm_no_ta

Re: [PR] Add flag to allow diabling creation of catalog tables [iceberg-python]

2024-09-17 Thread via GitHub
isc-patrick commented on code in PR #1155: URL: https://github.com/apache/iceberg-python/pull/1155#discussion_r1763403718 ## tests/catalog/test_sql.py: ## @@ -225,6 +237,93 @@ def test_creation_from_impl(catalog_name: str, warehouse: Path) -> None: ) +def confirm_no_ta

Re: [PR] Spark 3.5: Don't change table distribution when only altering local order [iceberg]

2024-09-17 Thread via GitHub
RussellSpitzer commented on code in PR #10774: URL: https://github.com/apache/iceberg/pull/10774#discussion_r1763468390 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala: ## @@ -226,11 +226,13 @@ cl

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-17 Thread via GitHub
RussellSpitzer commented on PR #11001: URL: https://github.com/apache/iceberg/pull/11001#issuecomment-2356281104 @jbonofre Have you had a chance to check this again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-17 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2356354267 Hi @sungwy I have already fixed the problem that ruff found. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-17 Thread via GitHub
jbonofre commented on PR #11001: URL: https://github.com/apache/iceberg/pull/11001#issuecomment-2356353914 Not yet but I don't want to hold this change. I propose to merge and as I'm working on dependencies cleanup and versions alignement I will check at that time. -- This is an automate

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-17 Thread via GitHub
RussellSpitzer commented on PR #11001: URL: https://github.com/apache/iceberg/pull/11001#issuecomment-2356365877 Ok, I think we are good to merge here. We can always revert if this ends up causing issues that we haven't forseen. -- This is an automated message from the Apache Git Service.

Re: [PR] Flink: Avoid metaspace memory leak by not registering ShutdownHook for ExecutorService in Flink [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11073: URL: https://github.com/apache/iceberg/pull/11073#discussion_r1763520712 ## core/src/main/java/org/apache/iceberg/util/ThreadPools.java: ## @@ -86,9 +86,18 @@ public static ExecutorService newWorkerPool(String namePrefix) { } pu

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-17 Thread via GitHub
RussellSpitzer merged PR #11001: URL: https://github.com/apache/iceberg/pull/11001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Build: switch to slf4j-simple 2.x for test implementation dependency [iceberg]

2024-09-17 Thread via GitHub
RussellSpitzer commented on PR #11001: URL: https://github.com/apache/iceberg/pull/11001#issuecomment-2356367551 Thanks @stevenzwu and @nastra and @jbonofre For Review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] AWS: Set better defaults for S3 retry behaviour [iceberg]

2024-09-17 Thread via GitHub
ookumuso commented on code in PR #11052: URL: https://github.com/apache/iceberg/pull/11052#discussion_r1763561503 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -393,6 +403,21 @@ public class S3FileIOProperties implements Serializable { */ p

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763567824 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
rdblue commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1763567963 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -578,6 +578,10 @@ public TableMetadata removeSnapshotsIf(Predicate removeIf) { return new Buil

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763568873 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
rdblue commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1763570028 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -964,6 +968,42 @@ private Builder(TableMetadata base) { this.sortOrdersById = Maps.newHashMap

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
rdblue commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1763571115 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1685,4 +1685,82 @@ public void testV3TimestampNanoTypeSupport() { ImmutableMap.of(),

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763576544 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Revert "Cache Manifest files" [iceberg-python]

2024-09-17 Thread via GitHub
sungwy commented on PR #1167: URL: https://github.com/apache/iceberg-python/pull/1167#issuecomment-2356453008 Hi @kevinjqliu thanks for putting this together. Would you mind sharing the verbose exception trace for this issue? Conceptually, I don't think this should result in an error becaus

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763584687 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763587936 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763610090 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763567824 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
nastra commented on code in PR #11151: URL: https://github.com/apache/iceberg/pull/11151#discussion_r1763617159 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1685,4 +1685,82 @@ public void testV3TimestampNanoTypeSupport() { ImmutableMap.of(),

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763610090 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763632736 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763610090 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-09-17 Thread via GitHub
ookumuso commented on code in PR #2: URL: https://github.com/apache/iceberg/pull/2#discussion_r1763671169 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3LocationProvider.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-09-17 Thread via GitHub
ookumuso commented on code in PR #2: URL: https://github.com/apache/iceberg/pull/2#discussion_r1763674102 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3LocationProvider.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-09-17 Thread via GitHub
ookumuso commented on PR #2: URL: https://github.com/apache/iceberg/pull/2#issuecomment-2356595248 > @ookumuso Overall, this looks like a great feature if this is better for S3 to repartition and distribute data, but it also seems like it would fit cleanly into the existing ObjectSt

Re: [PR] Core: Update metadata location without updating lastUpdatedMillis [iceberg]

2024-09-17 Thread via GitHub
rdblue commented on PR #11151: URL: https://github.com/apache/iceberg/pull/11151#issuecomment-2356600809 Thanks for the update! Looks good now. Please merge when tests are passing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

2024-09-17 Thread via GitHub
bryanck commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1763710592 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includes so

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

2024-09-17 Thread via GitHub
bryanck commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1763720315 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includes so

[I] Reference REST Catalog does not validate "to" identifier on rename table [iceberg]

2024-09-17 Thread via GitHub
haizhou-zhao opened a new issue, #11154: URL: https://github.com/apache/iceberg/issues/11154 ### Query engine Spark ### Question # Background Spark will pass `catalog` name to `renameTable` operations as part of its `to` identifier, and if that `catalog` name is not h

Re: [I] Adding RESTCatalog based Spark Smoke Test [iceberg]

2024-09-17 Thread via GitHub
haizhou-zhao commented on issue #11079: URL: https://github.com/apache/iceberg/issues/11079#issuecomment-2356671027 https://github.com/apache/iceberg/issues/11154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

2024-09-17 Thread via GitHub
danielcweeks commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763770630 ## core/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Revert "Cache Manifest files" [iceberg-python]

2024-09-17 Thread via GitHub
kevinjqliu commented on PR #1167: URL: https://github.com/apache/iceberg-python/pull/1167#issuecomment-2356727162 @sungwy thanks for following up on this. I added more details in the PR description. My suspicion is that this is due to the generators in [`read_manifest_list` ](https:/

Re: [PR] Revert "Cache Manifest files" [iceberg-python]

2024-09-17 Thread via GitHub
sungwy commented on PR #1167: URL: https://github.com/apache/iceberg-python/pull/1167#issuecomment-2356731020 > My suspicion is that this is due to the generators in [`read_manifest_list` ](https://github.com/apache/iceberg-python/blob/de47590c6ac4f507cb2337c20504a62c484339f9/pyiceberg/mani

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763800052 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] to_pandas(), to_arrow() fail because case_sensitive doesn't work if column in row_filter doesn't match the case even if case_sensitive is set to False in scan [iceberg-python]

2024-09-17 Thread via GitHub
leonidmakarovsky commented on issue #1177: URL: https://github.com/apache/iceberg-python/issues/1177#issuecomment-2356756573 Do I need to install the different pyiceberg version to confirm this? On Mon, Sep 16, 2024 at 2:07 PM Kevin Liu ***@***.***> wrote: > thanks for reportin

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763828430 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763830142 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763843986 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

2024-09-17 Thread via GitHub
pvary commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1763858710 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software

  1   2   >