Re: [I] OSSFileIO not compatible with aliyun-sdk-oss higher than 3.11.3 [iceberg]

2024-04-25 Thread via GitHub
769484623 commented on issue #9934: URL: https://github.com/apache/iceberg/issues/9934#issuecomment-2076510336 > Currently the version being used in Iceberg is > > https://github.com/apache/iceberg/blob/b714978583f5ba32ec34396cb82b0a8f100bf837/gradle/libs.versions.toml#L24 > >

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2076512208 I ran the integration test using my AWS account and the `testUpdateTable()` was successful. -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Tracking issues of iceberg-rust v0.3.0 [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on issue #348: URL: https://github.com/apache/iceberg-rust/issues/348#issuecomment-2076518611 > Hi, @Fokko About the read projection part, currently we can convert parquet files into arrow streams, but there are some limitations: it only support primitive types, and schema e

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1578992843 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579004695 ## crates/iceberg/src/io.rs: ## @@ -368,6 +368,9 @@ impl Storage { new_props.insert("root".to_string(), DEFAULT_ROOT_PATH.to_string()); match schem

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579004695 ## crates/iceberg/src/io.rs: ## @@ -368,6 +368,9 @@ impl Storage { new_props.insert("root".to_string(), DEFAULT_ROOT_PATH.to_string()); match schem

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2076564061 Also, I built the jars and manually tested on AWS Glue as a Spark runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Introduce hierarchical namespaces into SqlCatalog [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on PR #591: URL: https://github.com/apache/iceberg-python/pull/591#issuecomment-2076573394 > Should we allow table identifiers to have no namespace at all, like we do in Java? (pending @Fokko ) No, I don't think we should allow this. We also disallow this for the REST

[I] AWS: Updating Glue catalog table removes column descriptions [iceberg]

2024-04-25 Thread via GitHub
aajisaka opened a new issue, #10220: URL: https://github.com/apache/iceberg/issues/10220 ### Apache Iceberg version main (development) ### Query engine Spark ### Please describe the bug 🐞 In AWS Glue Catalog, user can set arbitrary descriptions to the table

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

2024-04-25 Thread via GitHub
ludlows commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1579048510 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc, M

Re: [PR] Test: Add test to partition on field with a dot [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #610: URL: https://github.com/apache/iceberg-python/pull/610#discussion_r1579050799 ## tests/integration/test_reads.py: ## @@ -473,6 +474,31 @@ def test_sanitize_character(catalog: Catalog) -> None: assert arrow_table.schema.names[0] == table_

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1579060905 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -135,6 +135,49 @@ public void createTestTable() throws IOException { commit(table, table.newAp

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1579059939 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -295,5 +343,6 @@ public void testValidatedOverwriteWithAppendSuccess() { .hasMessageStartin

Re: [PR] Refactor GlueCatalog's _commit_table [iceberg-python]

2024-04-25 Thread via GitHub
Fokko merged PR #653: URL: https://github.com/apache/iceberg-python/pull/653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1579083787 ## pyiceberg/catalog/__init__.py: ## @@ -710,6 +760,45 @@ def _get_updated_props_and_update_summary( return properties_update_summary, updated_properties

Re: [I] Is the "Emitting watermarks" new feature can't be used in flink sql? [iceberg]

2024-04-25 Thread via GitHub
pvary commented on issue #10219: URL: https://github.com/apache/iceberg/issues/10219#issuecomment-2076664228 @yeezychao: Do you happen to know, what is needed from the connector side to make this work? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1579101566 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc, Map

[PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
aajisaka opened a new pull request, #10221: URL: https://github.com/apache/iceberg/pull/10221 ## Problem TestGlueCatalogTable#testCreateTable failed by the following assertion error: ``` org.opentest4j.AssertionFailedError: expected: ["s3://iceberg-integ-68400161576

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579107619 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10221: URL: https://github.com/apache/iceberg/pull/10221#issuecomment-2076687909 Hi @nastra would you review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Sql catalog [iceberg-rust]

2024-04-25 Thread via GitHub
JanKaul commented on PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#issuecomment-2076689179 Thank you all for your helpful comments. I think the PR is ready for review again. @liurenjie1024 @sdd @odysa @ZENOTME @martin-g -- This is an automated message from the Apac

Re: [PR] Core: Use 'delete' / 'append' if OverwriteFiles only deletes/appends data files [iceberg]

2024-04-25 Thread via GitHub
nastra merged PR #10150: URL: https://github.com/apache/iceberg/pull/10150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] MinIO + Spark + hive metadata + iceberg format [iceberg]

2024-04-25 Thread via GitHub
rychu151 opened a new issue, #10222: URL: https://github.com/apache/iceberg/issues/10222 ### Query engine Spark ### Question Im trying to setup local develop env for my testing purposes using docker **Target is to save dataframe in a Iceberg format and Hive-metadat

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579198316 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579204149 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -174,20 +208,21 @@ private void flushDeletes() { } catch (IOException e) { set

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579208675 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -59,20 +60,58 @@ class SortedPosDeleteWriter implements FileWriter, DeleteWr Outpu

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579215950 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1579248989 ## crates/iceberg/src/scan.rs: ## @@ -158,8 +196,24 @@ impl TableScan { .await?; // Generate data file stream -let mut

Re: [I] Implement all functions of BoundPredicateVisitor for ManifestFilterVisitor [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #350: URL: https://github.com/apache/iceberg-rust/issues/350#issuecomment-2076877306 @s-akhtar-baig Assigned, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579255109 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [I] Implement the equality delete writer [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #341: URL: https://github.com/apache/iceberg-rust/issues/341#issuecomment-2076878340 Assigned to you, thanks @Dysprosium0626 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579255478 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -118,6 +118,10 @@ private DeleteWriteResult writeFileDeletes() throws IOEx

Re: [I] Test suite for manifest filtering [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #351: URL: https://github.com/apache/iceberg-rust/issues/351#issuecomment-2076879150 Assigned, thanks @s-akhtar-baig ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579259488 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -174,20 +208,21 @@ private void flushDeletes() { } catch (IOException e) { set

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on PR #10200: URL: https://github.com/apache/iceberg/pull/10200#issuecomment-2076931744 > After taking a closer look at `BaseTaskWriter`, I think we may have a correctness issue when encoding changes if the table contains multiple specs. Our current implementation of `BaseTa

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579292821 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,58 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

2024-04-25 Thread via GitHub
pvary commented on PR #10208: URL: https://github.com/apache/iceberg/pull/10208#issuecomment-2076940629 Are we good to go with this @stevenzwu ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579323906 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
nastra merged PR #10221: URL: https://github.com/apache/iceberg/pull/10221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579326259 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-04-25 Thread via GitHub
nastra commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2077042155 @epgif can you please address the test failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579368639 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579369755 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579372414 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579375646 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [I] How to move Iceberg table from one location to another [iceberg]

2024-04-25 Thread via GitHub
cccs-jc commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2077081125 @ksmatharoo, thank you for providing your code to replace the paths within the iceberg metadata. I've implemented a similar solution using Python's fastavro to handle Avro fil

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579401183 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579401733 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579402269 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

[I] Empty snapshot ID should be `Null` instead of `-1` [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko opened a new issue, #352: URL: https://github.com/apache/iceberg-rust/issues/352 This is an old bug from Java. Where the Snapshot was set to -1 instead of `None`: https://github.com/apache/iceberg-rust/blob/aba620900e99423bbd3fed969618e67e58a03a7b/crates/iceberg/src/spec/table_

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579406252 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579412576 ## crates/iceberg/src/transaction.rs: ## @@ -95,6 +104,42 @@ impl<'a> Transaction<'a> { Ok(self) } +/// Creates a fast append action. +pub fn f

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579420662 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579426812 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

[I] `field-id`'s missing in generated Avro files [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko opened a new issue, #353: URL: https://github.com/apache/iceberg-rust/issues/353 Looks like the `field-id` is missing in the Avro metadata generated by iceberg-rust. This should conform the spec: https://iceberg.apache.org/spec/#avro ``` avro-tools getschema snap-0-1-a0c0a37

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579449633 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579449633 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [I] `field-id`'s missing in generated Avro files [iceberg-rust]

2024-04-25 Thread via GitHub
s-akhtar-baig commented on issue #353: URL: https://github.com/apache/iceberg-rust/issues/353#issuecomment-2077159481 @Fokko, can you please assign this to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] Correct names in the ManifestList [iceberg-rust]

2024-04-25 Thread via GitHub
s-akhtar-baig commented on issue #354: URL: https://github.com/apache/iceberg-rust/issues/354#issuecomment-2077159735 @Fokko, can you please assign this to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] `field-id`'s missing in generated Avro files [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on issue #353: URL: https://github.com/apache/iceberg-rust/issues/353#issuecomment-2077166551 @s-akhtar-baig Certainly 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579466191 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

2024-04-25 Thread via GitHub
bryanck commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1579467601 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/IcebergWriterFactoryTest.java: ## @@ -83,4 +90,26 @@ public void testAutoCreateTable(bool

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579472707 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579520268 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10221: URL: https://github.com/apache/iceberg/pull/10221#issuecomment-2077332098 Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Sql catalog [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1579259139 ## crates/catalog/sql/Cargo.toml: ## @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579601294 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579654225 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Update 1.5.1 release notes [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #10224: URL: https://github.com/apache/iceberg/pull/10224#discussion_r1579716696 ## site/docs/releases.md: ## @@ -73,6 +73,21 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.1 release +Apache Iceb

Re: [PR] Update 1.5.1 release notes [iceberg]

2024-04-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #10224: URL: https://github.com/apache/iceberg/pull/10224#discussion_r1579724014 ## site/docs/releases.md: ## @@ -73,6 +73,21 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.1 release +A

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
Xuanwo commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579739285 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

2024-04-25 Thread via GitHub
viirya commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2077637850 Thanks @liurenjie1024. The roadmaps doc looks good to me. I added a few items under DataFusion integration. Feel free to modify it. Thanks. -- This is an automated message from t

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579755826 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Update 1.5.1 release notes [iceberg]

2024-04-25 Thread via GitHub
nastra commented on PR #10224: URL: https://github.com/apache/iceberg/pull/10224#issuecomment-2077658431 LGTM, thanks @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Update 1.5.1 release notes [iceberg]

2024-04-25 Thread via GitHub
nastra merged PR #10224: URL: https://github.com/apache/iceberg/pull/10224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2077663647 > I've left some comment to improve, but it looks great! I'll invite datafusion community to help review. @liurenjie1024 Thanks for the review. I fixed most of the

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
tshauck commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579818649 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579827548 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-25 Thread via GitHub
stevenzwu commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1579860068 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-04-25 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2077790169 > @epgif can you please address the test failures? Done. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

2024-04-25 Thread via GitHub
stevenzwu commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1579871219 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -40,24 +44,27 @@ import org.apache.iceberg.FileFormat; im

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579883806 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579897701 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579919751 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579930294 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579940862 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579945951 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1580019899 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-25 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580024955 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

[I] byte and short types in spark no longer auto coerce to int32 [iceberg]

2024-04-25 Thread via GitHub
jkolash opened a new issue, #10225: URL: https://github.com/apache/iceberg/issues/10225 ### Apache Iceberg version 1.5.0 ### Query engine Spark ### Please describe the bug 🐞 The removal of the code ```java private static PrimitiveWriter ints(DataT

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

2024-04-25 Thread via GitHub
sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580038792 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +240,25 @@ impl TableScan { .boxed()) } +fn create_manifest_evaluator( +id: i32, +schema:

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

2024-04-25 Thread via GitHub
sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580039092 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +240,25 @@ impl TableScan { .boxed()) } +fn create_manifest_evaluator( +id: i32, +schema:

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

2024-04-25 Thread via GitHub
sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580049448 ## crates/iceberg/src/scan.rs: ## @@ -158,8 +196,24 @@ impl TableScan { .await?; // Generate data file stream -let mut entries =

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

2024-04-25 Thread via GitHub
Fokko commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078097905 Hey @jkolash Thanks for reporting this. The behavior should stay the same, due to the logic here: https://github.com/apache/iceberg/pull/9440/files#diff-8ac59cbdbcc60cc0c55805

  1   2   >