Re: [PR] Tests: Unify the test catalog setting [iceberg-python]

2024-04-16 Thread via GitHub
frankliee commented on code in PR #609: URL: https://github.com/apache/iceberg-python/pull/609#discussion_r1566818851 ## tests/conftest.py: ## @@ -2144,3 +2144,31 @@ def arrow_table_with_only_nulls(pa_schema: "pa.Schema") -> "pa.Table": import pyarrow as pa return p

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1566850598 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -391,17 +391,16 @@ public void createTable(ObjectPath tablePath, CatalogBaseTable

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1566852990 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -391,17 +391,16 @@ public void createTable(ObjectPath tablePath, CatalogBaseTable

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1566855387 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -391,17 +391,16 @@ public void createTable(ObjectPath tablePath, CatalogBaseTable

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
nastra commented on PR #10112: URL: https://github.com/apache/iceberg/pull/10112#issuecomment-2058425533 > @nastra: Any comments? I would like to merge this soon, as any merge to Flink code path will make this PR stale, and @rodmeneses needs to recreate the whole PR. > > Thanks,

[PR] Add `BoundPredicateVisitor` (alternate version) [iceberg-rust]

2024-04-16 Thread via GitHub
sdd opened a new pull request, #334: URL: https://github.com/apache/iceberg-rust/pull/334 Alternative implementation for https://github.com/apache/iceberg-rust/pull/320, using per-operator visitor methods -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

2024-04-16 Thread via GitHub
nastra commented on PR #10124: URL: https://github.com/apache/iceberg/pull/10124#issuecomment-2058435530 @mrcnc looks like there's a merge conflict. Can you rebase please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
sdd commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1566871505 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,317 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] Support partial deletes [iceberg-python]

2024-04-16 Thread via GitHub
Fokko commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1566870388 ## pyiceberg/io/pyarrow.py: ## @@ -1912,3 +1920,55 @@ def _get_parquet_writer_kwargs(table_properties: Properties) -> Dict[str, Any]: default=TableProp

[PR] add `InclusiveProjection` Visitor (alternate version) [iceberg-rust]

2024-04-16 Thread via GitHub
sdd opened a new pull request, #335: URL: https://github.com/apache/iceberg-rust/pull/335 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058445375 @kevinjqliu It is strange to me that in PyArrow there is [pa.unify_schemas(...)](https://arrow.apache.org/docs/python/generated/pyarrow.unify_schemas.html) which is able (I do

Re: [PR] Core, Spark: Use 'delete' if RowDelta only has delete files [iceberg]

2024-04-16 Thread via GitHub
nastra merged PR #10123: URL: https://github.com/apache/iceberg/pull/10123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Delete using Merge-on-Read sets `OVERWRITE` while `DELETE` is expected [iceberg]

2024-04-16 Thread via GitHub
nastra closed issue #10122: Delete using Merge-on-Read sets `OVERWRITE` while `DELETE` is expected URL: https://github.com/apache/iceberg/issues/10122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058456159 > One thing I wonder is if PyIceberg can handle schema evolution of nested structs. Looks like it can. From https://py.iceberg.apache.org/api/#add-column: ``` with

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
sdd commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1566902353 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,317 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] Core, Spark: Use 'delete' if RowDelta only has delete files [iceberg]

2024-04-16 Thread via GitHub
nastra commented on PR #10123: URL: https://github.com/apache/iceberg/pull/10123#issuecomment-2058468333 thanks for the reviews @amogh-jahagirdar and @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-16 Thread via GitHub
pvary opened a new pull request, #10151: URL: https://github.com/apache/iceberg/pull/10151 FileIO implementation which uses Flink embedded filesystem implementations. This allows the user to use the Flink pluggable FileSystems: https://nightlies.apache.org/flink/flink-docs-release-1.19

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058470428 BTW: Found some explaination why merge of Arrow tables with different schemas is not possible: https://github.com/apache/arrow/issues/35424 The reason looks weired, but yes,

Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]

2024-04-16 Thread via GitHub
sannaroby commented on issue #10147: URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2058495395 Hi @pvary, thanks for your reply. We're using the HASH distribution mode and this is an extract from our flink job: ``` SingleOutputStreamOperator mainFunction = env.

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566934826 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTha

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566941616 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -83,9 +83,32 @@ public void testBuilderDoesNotCreateInvalidObjects() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566941917 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -105,9 +105,32 @@ public void testBuilderDoesNotCreateInvalidObjects() {

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
wForget commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1566943822 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java: ## @@ -275,18 +275,20 @@ public Table alterTable(Identifier ident, TableChange...

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566944912 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,36 @@ public void closeCatalog() throws Exception { } } + @Test + p

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566933302 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -34,7 +34,7 @@ public class TestListNamespacesResponse extends RequestRes

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566937039 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTha

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566939950 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTha

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566939429 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTha

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566946785 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

[PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 (#101111) [iceberg]

2024-04-16 Thread via GitHub
jbonofre opened a new pull request, #10152: URL: https://github.com/apache/iceberg/pull/10152 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 (#101111) [iceberg]

2024-04-16 Thread via GitHub
jbonofre commented on PR #10152: URL: https://github.com/apache/iceberg/pull/10152#issuecomment-2058580254 @nastra this is the backport to the `1.5.x` branch (in preparation for 1.5.1 release). -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

2024-04-16 Thread via GitHub
Fokko commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2058636547 @marvinlanhenke Sorry for being late to the party here. Appending a file is rather straightforward, but all the conditions must be met. This is the high-level way of appending a fil

Re: [I] Doubts about the types supported by Iceberg, Not in line with expectations [iceberg]

2024-04-16 Thread via GitHub
Fokko commented on issue #10153: URL: https://github.com/apache/iceberg/issues/10153#issuecomment-2058638829 @madeirak Thanks for reaching out here. Which version of Spark are you using? -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] flink:FlinkSink support dynamically changed schema [iceberg]

2024-04-16 Thread via GitHub
Ruees commented on issue #4190: URL: https://github.com/apache/iceberg/issues/4190#issuecomment-2058647346 > @leichangqing You can refer to the last two commits of my branch https://github.com/lintingbin2009/iceberg/tree/flink-sink-dynamically-change. We have put this part of the code in ou

Re: [I] Doubts about the types supported by Iceberg, Not in line with expectations [iceberg]

2024-04-16 Thread via GitHub
madeirak closed issue #10153: Doubts about the types supported by Iceberg, Not in line with expectations URL: https://github.com/apache/iceberg/issues/10153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
ajantha-bhat commented on issue #10121: URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058678837 how is the table partitioned? Can you do describe table or show create table to show the partition scheme? Also, was there any partition evolution? Are you using V1 Iceberg ta

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567071915 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567077122 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566939429 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTha

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567077122 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1566940267 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,36 @@ public void closeCatalog() throws Exception { } } + @Test + p

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567097286 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List listTab

Re: [I] Remove extraneous trailing slash in table location [iceberg-python]

2024-04-16 Thread via GitHub
Vitalii0-o commented on issue #606: URL: https://github.com/apache/iceberg-python/issues/606#issuecomment-2058724052 Traceback (most recent call last): File "/usr/local/airflow/.local/lib/python3.11/site-packages/dlt/destinations/sql_client.py", line 242, in _wrap_gen return (yi

Re: [I] Remove extraneous trailing slash in table location [iceberg-python]

2024-04-16 Thread via GitHub
Vitalii0-o commented on issue #606: URL: https://github.com/apache/iceberg-python/issues/606#issuecomment-2058726484 This error occurred to me about a week ago, but there were no changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
madeirak commented on issue #10121: URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058730627 > how is the table partitioned? Can you do describe table or show create table to show the partition scheme? Also, was there any partition evolution? Are you using V1 Iceberg tabl

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
ajantha-bhat commented on issue #10121: URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058743091 I think Iceberg v1 tables has a concept of void transform and it will keep the dropped partition still as a void after partition evolution. Try changing the table defau

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
ajantha-bhat commented on issue #10121: URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058745161 Lastly I think there is no way to delete empty folders in Iceberg. Only files are tracked by Iceberg metadata. So, GC operations only clean up the files. -- This is an aut

Re: [I] Iceberg Spark Extensions conflict with Paimon [iceberg]

2024-04-16 Thread via GitHub
ajantha-bhat commented on issue #10143: URL: https://github.com/apache/iceberg/issues/10143#issuecomment-2058746453 can you close this if it is a duplicate of https://github.com/apache/paimon/issues/3212 ? -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
Fokko commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1567138774 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,317 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contrib

Re: [I] Flaky spark-3.3-scala-2.13-java-8 tests due to `No space left on device` [iceberg]

2024-04-16 Thread via GitHub
manuzhang commented on issue #10040: URL: https://github.com/apache/iceberg/issues/10040#issuecomment-2058792129 Can this be related to https://issues.apache.org/jira/browse/SPARK-39195? @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
madeirak commented on issue #10121: URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058796187 > Lastly I think there is no way to delete empty folders in Iceberg. Only files are tracked by Iceberg metadata. So, GC operations only clean up the files. thx set version

Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]

2024-04-16 Thread via GitHub
madeirak closed issue #10121: Can Iceberg old partition with no data be deleted? URL: https://github.com/apache/iceberg/issues/10121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Tests: Unify the test catalog setting [iceberg-python]

2024-04-16 Thread via GitHub
frankliee commented on code in PR #609: URL: https://github.com/apache/iceberg-python/pull/609#discussion_r1566818851 ## tests/conftest.py: ## @@ -2144,3 +2144,31 @@ def arrow_table_with_only_nulls(pa_schema: "pa.Schema") -> "pa.Table": import pyarrow as pa return p

Re: [PR] Tests: Unify the test catalog setting [iceberg-python]

2024-04-16 Thread via GitHub
frankliee commented on code in PR #609: URL: https://github.com/apache/iceberg-python/pull/609#discussion_r1566818851 ## tests/conftest.py: ## @@ -2144,3 +2144,31 @@ def arrow_table_with_only_nulls(pa_schema: "pa.Schema") -> "pa.Table": import pyarrow as pa return p

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

2024-04-16 Thread via GitHub
deniskuzZ commented on PR #10148: URL: https://github.com/apache/iceberg/pull/10148#issuecomment-2058916040 AFAIK, autogater doesn't even work in Hive. After some operations like insert, we issue an extra stats update task that persists column stats changes either to the HMS or puffin file

Re: [I] spark.table() raises warn: Unclosed S3FileIO instance in HadoopTableOperations [iceberg]

2024-04-16 Thread via GitHub
KingLommel commented on issue #10145: URL: https://github.com/apache/iceberg/issues/10145#issuecomment-2058941436 Thanks @ajantha-bhat. As soon as I have time to dive deep into this I will give it a try! -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1567303582 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,317 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add `BoundPredicateVisitor` (alternate version) [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on PR #334: URL: https://github.com/apache/iceberg-rust/pull/334#issuecomment-2059023496 I'll merge this given we have reached some consensus [here](https://github.com/apache/iceberg-rust/pull/320#discussion_r1565780931), thanks @Fokko for review, and @sdd for this e

Re: [PR] Add `BoundPredicateVisitor` (alternate version) [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 merged PR #334: URL: https://github.com/apache/iceberg-rust/pull/334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Lazily compute & cache hashCode in CharSequenceWrapper [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #10023: URL: https://github.com/apache/iceberg/pull/10023#discussion_r1567311373 ## api/src/main/java/org/apache/iceberg/util/CharSequenceWrapper.java: ## @@ -29,13 +29,16 @@ public static CharSequenceWrapper wrap(CharSequence seq) { } priv

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 closed pull request #320: Add `BoundPredicateVisitor` trait URL: https://github.com/apache/iceberg-rust/pull/320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#issuecomment-2059024512 Close by #334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on code in PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#discussion_r1565832014 ## crates/catalog/glue/src/utils.rs: ## @@ -151,6 +205,65 @@ pub(crate) fn validate_namespace(namespace: &NamespaceIdent) -> Result { Ok(name) } +///

Re: [I] Remove `unwrap()` in `ManifestListWriter.close()` [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on issue #177: URL: https://github.com/apache/iceberg-rust/issues/177#issuecomment-2059068941 > Is this issue still valid given #185 ? This seems addressed. Apologies for the noise, surfing good first issues for something to get started on. @tabmatfournier Yeah

Re: [I] Remove `unwrap()` in `ManifestListWriter.close()` [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 closed issue #177: Remove `unwrap()` in `ManifestListWriter.close()` URL: https://github.com/apache/iceberg-rust/issues/177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Lazily compute & cache hashCode in CharSequenceWrapper [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #10023: URL: https://github.com/apache/iceberg/pull/10023#discussion_r1567311373 ## api/src/main/java/org/apache/iceberg/util/CharSequenceWrapper.java: ## @@ -29,13 +29,16 @@ public static CharSequenceWrapper wrap(CharSequence seq) { } priv

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
marvinlanhenke commented on code in PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#discussion_r1567401176 ## crates/catalog/glue/src/utils.rs: ## @@ -151,6 +205,65 @@ pub(crate) fn validate_namespace(namespace: &NamespaceIdent) -> Result { Ok(name) } +///

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
marvinlanhenke commented on code in PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#discussion_r1567401176 ## crates/catalog/glue/src/utils.rs: ## @@ -151,6 +205,65 @@ pub(crate) fn validate_namespace(namespace: &NamespaceIdent) -> Result { Ok(name) } +///

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
marvinlanhenke commented on code in PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#discussion_r1567403891 ## crates/catalog/glue/src/catalog.rs: ## @@ -310,31 +327,282 @@ impl Catalog for GlueCatalog { Ok(table_list) } +/// Creates a new table

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
marvinlanhenke commented on code in PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#discussion_r1567406955 ## crates/catalog/glue/src/catalog.rs: ## @@ -310,31 +327,282 @@ impl Catalog for GlueCatalog { Ok(table_list) } +/// Creates a new table

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

2024-04-16 Thread via GitHub
marvinlanhenke commented on PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#issuecomment-2059160129 > Thanks for @marvinlanhenke for this great pr, it looks great! Sorry for late reply, I have been busy lately. @liurenjie1024 No worries, and thanks for the review.

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 (#101111) [iceberg]

2024-04-16 Thread via GitHub
nastra merged PR #10152: URL: https://github.com/apache/iceberg/pull/10152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2059202270 I think to implement appending data file, there are two main tasks: 1. Implement transaction api to append data file 2. Implement file writer to write record batches

Re: [PR] Sql catalog [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#issuecomment-2059212759 cc @JanKaul Is this pr ready for review or you need to do more updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-16 Thread via GitHub
liurenjie1024 commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1567468384 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) }

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
rodmeneses commented on PR #10112: URL: https://github.com/apache/iceberg/pull/10112#issuecomment-2059306340 > > @nastra: Any comments? I would like to merge this soon, as any merge to Flink code path will make this PR stale, and @rodmeneses needs to recreate the whole PR. > > Thanks,

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
pvary commented on PR #10112: URL: https://github.com/apache/iceberg/pull/10112#issuecomment-2059345426 > Why do we remove Flink 1.1.6 in this PR? @manuzhang: This is how we usually do these changes. We support the 3 last version of Flink, so when we add a new version, we remove the o

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
pvary merged PR #10112: URL: https://github.com/apache/iceberg/pull/10112 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Tests: Unify the test catalog setting [iceberg-python]

2024-04-16 Thread via GitHub
HonahX merged PR #609: URL: https://github.com/apache/iceberg-python/pull/609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-16 Thread via GitHub
pvary commented on PR #10112: URL: https://github.com/apache/iceberg/pull/10112#issuecomment-2059391305 Merged to main. Thanks for the PR @rodmeneses and @nastra for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Removes Flink version 1.16 [iceberg]

2024-04-16 Thread via GitHub
rodmeneses commented on PR #10154: URL: https://github.com/apache/iceberg/pull/10154#issuecomment-2059418597 cc: @pvary @nastra please take a look. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] JDBC Catalog:Change SQL like escape character from '\\' to '!' [iceberg]

2024-04-16 Thread via GitHub
nastra commented on PR #9407: URL: https://github.com/apache/iceberg/pull/9407#issuecomment-2059442673 fixes https://github.com/apache/iceberg/issues/10056 @xuchuanqiu can you rebase this one please to fix the merge conflicts? -- This is an automated message from the Apache Git Serv

Re: [PR] JDBC Catalog:Change SQL like escape character from '\\' to '!' [iceberg]

2024-04-16 Thread via GitHub
jbonofre commented on code in PR #9407: URL: https://github.com/apache/iceberg/pull/9407#discussion_r1567618729 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -136,7 +136,7 @@ final class JdbcUtil { + TABLE_NAMESPACE + " = ? OR "

Re: [PR] JDBC Catalog:Change SQL like escape character from '\\' to '!' [iceberg]

2024-04-16 Thread via GitHub
jbonofre commented on PR #9407: URL: https://github.com/apache/iceberg/pull/9407#issuecomment-2059448849 If @xuchuanqiu doesn't have the bandwidth, I can help on this one. If it works with MySQL, SQLlite and PostgreSQL, it's a nice fix to include in Iceberg 1.5.1. -- This is an automated

Re: [PR] Removes Flink version 1.16 [iceberg]

2024-04-16 Thread via GitHub
pvary merged PR #10154: URL: https://github.com/apache/iceberg/pull/10154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
stevenzwu commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567629192 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Removes Flink version 1.16 [iceberg]

2024-04-16 Thread via GitHub
pvary commented on PR #10154: URL: https://github.com/apache/iceberg/pull/10154#issuecomment-2059456318 Thanks for the PR @rodmeneses and @nastra for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
stevenzwu commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567629965 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Removes Flink version 1.16 [iceberg]

2024-04-16 Thread via GitHub
Fokko commented on PR #10154: URL: https://github.com/apache/iceberg/pull/10154#issuecomment-2059523299 Thanks @rodmeneses for working on this. Can you [update the table here](https://github.com/apache/iceberg/blob/main/site/docs/multi-engine-support.md#apache-flink) as well? -- This is

Re: [PR] Removes Flink version 1.16 [iceberg]

2024-04-16 Thread via GitHub
rodmeneses commented on PR #10154: URL: https://github.com/apache/iceberg/pull/10154#issuecomment-2059543391 @Fokko Thanks for the reminder. Here's the PR: https://github.com/apache/iceberg/pull/10155 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Updates flink versioning information in our docs [iceberg]

2024-04-16 Thread via GitHub
Fokko commented on code in PR #10155: URL: https://github.com/apache/iceberg/pull/10155#discussion_r1567700280 ## site/docs/multi-engine-support.md: ## @@ -83,15 +83,16 @@ Users should continuously upgrade their Flink version to stay up-to-date. | Version | Lifecycle Stage

[I] Iceberg Spark streaming skips rows of data [iceberg]

2024-04-16 Thread via GitHub
cccs-jc opened a new issue, #10156: URL: https://github.com/apache/iceberg/issues/10156 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When using spark readStream the option `stream-from-timestamp` is u

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-16 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1567725081 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567730841 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List listTa

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567734951 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -34,7 +34,7 @@ public class TestListNamespacesResponse extends RequestRe

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-16 Thread via GitHub
sdd closed pull request #321: add `InclusiveProjection` Visitor URL: https://github.com/apache/iceberg-rust/pull/321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-16 Thread via GitHub
sdd commented on PR #321: URL: https://github.com/apache/iceberg-rust/pull/321#issuecomment-2059636629 Closing this one in favour of the alternate PR based on the BoundPredicateVisitor design that got merged (https://github.com/apache/iceberg-rust/pull/335) -- This is an automated messag

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567745943 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -36,7 +36,7 @@ public class TestListTablesResponse extends RequestResponseTe

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567748849 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTh

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567755548 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTh

  1   2   >