Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869997024 Seems not resolved: ``` dig @114.114.114.114 rust.iceberg.apache.org ; <<>> DiG 9.10.6 <<>> @114.114.114.114 rust.iceberg.apache.org ; (1 server found) ;; g

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
Xuanwo commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869920279 > I can't reach to 8.8.8.8🥵 Please try `114.114.114.114` or `223.5.5.5` -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869911819 I can't reach to 8.8.8.8🥵 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
Xuanwo commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869904668 > Seems dns record added: > > ``` > dig rust.iceberg.apache.org > > ; <<>> DiG 9.10.6 <<>> rust.iceberg.apache.org > ;; global options: +cmd > ;; Got answer:

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869902638 But I can't open the page -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1869902465 Seems dns record added: ``` dig rust.iceberg.apache.org ; <<>> DiG 9.10.6 <<>> rust.iceberg.apache.org ;; global options: +cmd ;; Got answer: ;; ->>HEADE

Re: [PR] feat: Add roadmap and features status in README.md [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on PR #134: URL: https://github.com/apache/iceberg-rust/pull/134#issuecomment-1869894156 cc @Fokko @ZENOTME Any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

2023-12-26 Thread via GitHub
zinking commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436689974 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

2023-12-26 Thread via GitHub
zinking commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436689327 ## core/src/main/java/org/apache/iceberg/deletes/DeleteGranularity.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [I] dropDeleteFilesOlderthan should be partition level instead of table level [iceberg]

2023-12-26 Thread via GitHub
zinking commented on issue #9383: URL: https://github.com/apache/iceberg/issues/9383#issuecomment-1869871515 @RussellSpitzer any comments ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] DecimalType declaration check missing proper assertions [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] closed issue #7420: DecimalType declaration check missing proper assertions URL: https://github.com/apache/iceberg/issues/7420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] DecimalType declaration check missing proper assertions [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] commented on issue #7420: URL: https://github.com/apache/iceberg/issues/7420#issuecomment-1869834335 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Inconsistent API for remove_orphan_files and DeleteOrphanFiles [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] commented on issue #7480: URL: https://github.com/apache/iceberg/issues/7480#issuecomment-1869834322 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Is is possible to control the number of partitions (groups) for compaction ? [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] closed issue #7506: Is is possible to control the number of partitions (groups) for compaction ? URL: https://github.com/apache/iceberg/issues/7506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Is is possible to control the number of partitions (groups) for compaction ? [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] commented on issue #7506: URL: https://github.com/apache/iceberg/issues/7506#issuecomment-1869834307 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Inconsistent API for remove_orphan_files and DeleteOrphanFiles [iceberg]

2023-12-26 Thread via GitHub
github-actions[bot] closed issue #7480: Inconsistent API for remove_orphan_files and DeleteOrphanFiles URL: https://github.com/apache/iceberg/issues/7480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Glue catalog commit table [iceberg-python]

2023-12-26 Thread via GitHub
pdames commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1436639053 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab

Re: [PR] Glue catalog commit table [iceberg-python]

2023-12-26 Thread via GitHub
pdames commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1436629843 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab

Re: [PR] Glue catalog commit table [iceberg-python]

2023-12-26 Thread via GitHub
pdames commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1436629843 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab

Re: [PR] Glue catalog commit table [iceberg-python]

2023-12-26 Thread via GitHub
pdames commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1436629843 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab

Re: [PR] Glue catalog commit table [iceberg-python]

2023-12-26 Thread via GitHub
pdames commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1436629843 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab

[PR] AWS: Add S3 Access Grants Integration [iceberg]

2023-12-26 Thread via GitHub
adnanhemani opened a new pull request, #9385: URL: https://github.com/apache/iceberg/pull/9385 This PR allows us to add support for S3 Access Grants (S3AG) to S3FileIO through adding a new SDK Plugin (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/kafkaconnect/mod

Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

2023-12-26 Thread via GitHub
aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436606528 ## core/src/main/java/org/apache/iceberg/io/FanoutPositionOnlyDeleteWriter.java: ## @@ -49,21 +51,32 @@ public FanoutPositionOnlyDeleteWriter( OutputFileFact

Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

2023-12-26 Thread via GitHub
aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436606249 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "h

[PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

2023-12-26 Thread via GitHub
aokolnychyi opened a new pull request, #9384: URL: https://github.com/apache/iceberg/pull/9384 This PR adds support for file and partition delete granularity, allowing users to pick between the two. Under partition granularity, delete writers are allowed to group deletes for differen

Re: [PR] Core: Add KLL Datasketch and Hive ColumnStatisticsObj as standard blo… [iceberg]

2023-12-26 Thread via GitHub
ZacBlanco commented on PR #8202: URL: https://github.com/apache/iceberg/pull/8202#issuecomment-1869661970 Any progress update here? It would be great to get the blessing from the Iceberg community for these as supported puffin blob types -- This is an automated message from the Apache Git

[I] dropDeleteFilesOlderthan should be partition level instead of table level [iceberg]

2023-12-26 Thread via GitHub
zinking opened a new issue, #9383: URL: https://github.com/apache/iceberg/issues/9383 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 ``` public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Spark 3.5: Migrate remaining tests in source directory to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
chinmay-bhat commented on code in PR #9380: URL: https://github.com/apache/iceberg/pull/9380#discussion_r1436475714 ## data/src/test/java/org/apache/iceberg/data/ParameterizedDeleteReadTest.java: ## @@ -0,0 +1,591 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Flink: Create CatalogTestBase for migration to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1869562291 @nastra Please find this new PR. https://github.com/apache/iceberg/pull/9381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Spark Streaming: Fix clobbering of files across streaming epochs [iceberg]

2023-12-26 Thread via GitHub
VeeraswamyGatta commented on code in PR #9255: URL: https://github.com/apache/iceberg/pull/9255#discussion_r1436455766 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -673,11 +673,11 @@ public DataWriter createWriter(int partitionId, lon

[PR] Adding Junit 5 conversion and AssertJ style for TestFlinkCatalogTable… [iceberg]

2023-12-26 Thread via GitHub
vinitpatni opened a new pull request, #9381: URL: https://github.com/apache/iceberg/pull/9381 - Added junit 5 conversion and AssertJ style for TestFlinkCatalogTable and TestFlinkMetadataTable -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Flink: Create CatalogTestBase for migration to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
nastra commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1869530035 Just create a new PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Flink: Create CatalogTestBase for migration to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1869525715 @nastra Thanks for merging my changes. I have also completed Junit 5 conversion and AssertJ style for TestFlinkCatalogTable and TestFlinkMetadataTable. Should I created Separate PR or

Re: [PR] website: Fix typo in book.toml [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko commented on PR #136: URL: https://github.com/apache/iceberg-rust/pull/136#issuecomment-1869506307 @Xuanwo no problem! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] website: Fix typo in book.toml [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko merged PR #136: URL: https://github.com/apache/iceberg-rust/pull/136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

2023-12-26 Thread via GitHub
Xuanwo opened a new issue, #137: URL: https://github.com/apache/iceberg-rust/issues/137 `rust.iceberg.apache.org` is not resolved, do we need to submit a ticket? ```shell :) drill rust.iceberg.apache.org ;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 27319 ;; flags: qr rd

Re: [PR] website: Fix typo in book.toml [iceberg-rust]

2023-12-26 Thread via GitHub
Xuanwo commented on PR #136: URL: https://github.com/apache/iceberg-rust/pull/136#issuecomment-1869490104 cc @Fokko, I made a typo in the config, sorry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] website: Fix typo in book.toml [iceberg-rust]

2023-12-26 Thread via GitHub
Xuanwo opened a new pull request, #136: URL: https://github.com/apache/iceberg-rust/pull/136 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [I] write.target-file-size-bytes isn't respected when writing data [iceberg]

2023-12-26 Thread via GitHub
atifiu commented on issue #8729: URL: https://github.com/apache/iceberg/issues/8729#issuecomment-1869488679 @paulpaul1076 It could be really great if you can add some explanation which you have understood regarding this as it might benefit others also. -- This is an automated message from

Re: [I] How to improve write speed for data in the same partition? [iceberg]

2023-12-26 Thread via GitHub
atifiu commented on issue #9330: URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1869464036 @xuchang-66 @TechTinkerer42 I have tried to test this with `write.distribution-mode = none` and it can indeed help to improve the performance but can introduce the problem of small fi

Re: [PR] Flink: Create CatalogTestBase for migration to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
nastra merged PR #9364: URL: https://github.com/apache/iceberg/pull/9364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
chinmay-bhat commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1436382864 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java: ## @@ -94,7 +95,7 @@ public class TestCreateActions extends SparkCatalog

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1436381154 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java: ## @@ -516,17 +527,16 @@ public void testProperties() throws Exception {

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1436380170 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java: ## @@ -94,7 +95,7 @@ public class TestCreateActions extends SparkCatalogTestBa

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1436379721 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java: ## @@ -117,47 +119,48 @@ public static Object[][] parameters() {

Re: [PR] init writer framework [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1869428917 You can find how to write class diagram uml here: https://plantuml.com/class-diagram -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] init writer framework [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1869426878 I have worked together with @ZENOTME to implement such a class hierarchy in [icelake](https://github.com/icelake-io/icelake), so I'm familiar with this. I feel that maybe a UML d

Re: [PR] Spark 3.5: Migrate remaining tests in source directory to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
chinmay-bhat commented on code in PR #9380: URL: https://github.com/apache/iceberg/pull/9380#discussion_r1436375033 ## data/src/test/java/org/apache/iceberg/data/ParameterizedDeleteReadTest.java: ## @@ -0,0 +1,591 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Spark 3.5: Migrate remaining tests in source directory to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
chinmay-bhat commented on code in PR #9380: URL: https://github.com/apache/iceberg/pull/9380#discussion_r1436374870 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -57,39 +62,52 @@ import org.apache.spark.sql.connector.read.SupportsPu

Re: [PR] Write support [iceberg-python]

2023-12-26 Thread via GitHub
Fokko commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1436371090 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2004,158 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -1

Re: [PR] Spark 3.5: Migrate remaining tests in source directory to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
nastra commented on code in PR #9380: URL: https://github.com/apache/iceberg/pull/9380#discussion_r1436370156 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -57,39 +62,52 @@ import org.apache.spark.sql.connector.read.SupportsPushDown

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
chinmay-bhat commented on PR #9367: URL: https://github.com/apache/iceberg/pull/9367#issuecomment-1869415869 rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Spark 3.5: Migrate remaining tests in source directory to JUnit5 [iceberg]

2023-12-26 Thread via GitHub
nastra commented on code in PR #9380: URL: https://github.com/apache/iceberg/pull/9380#discussion_r1436367845 ## data/src/test/java/org/apache/iceberg/data/ParameterizedDeleteReadTest.java: ## @@ -0,0 +1,591 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

2023-12-26 Thread via GitHub
nastra commented on PR #9367: URL: https://github.com/apache/iceberg/pull/9367#issuecomment-1869413632 @chinmay-bhat can you please rebase against latest main? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] init writer framework [iceberg-rust]

2023-12-26 Thread via GitHub
ZENOTME commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1869402828 https://github.com/apache/iceberg-rust/assets/43447882/7b830b61-e898-49e6-85f4-eb11a47455b0";> I draw a diagram to demonstrate the design of writer. Please let me know if it stil

Re: [PR] feat: Introduce basic file scan planning. [iceberg-rust]

2023-12-26 Thread via GitHub
liurenjie1024 commented on code in PR #129: URL: https://github.com/apache/iceberg-rust/pull/129#discussion_r1436354564 ## crates/iceberg/Cargo.toml: ## @@ -62,4 +62,5 @@ uuid = { workspace = true } [dev-dependencies] pretty_assertions = { workspace = true } tempfile = { work

Re: [PR] feat: Introduce basic file scan planning. [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko commented on code in PR #129: URL: https://github.com/apache/iceberg-rust/pull/129#discussion_r1436336954 ## crates/iceberg/Cargo.toml: ## @@ -62,4 +62,5 @@ uuid = { workspace = true } [dev-dependencies] pretty_assertions = { workspace = true } tempfile = { workspace =

Re: [I] Terms, Types and Expressions [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko closed issue #22: Terms, Types and Expressions URL: https://github.com/apache/iceberg-rust/issues/22 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] feat: Expression system. [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko merged PR #132: URL: https://github.com/apache/iceberg-rust/pull/132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat: Expression system. [iceberg-rust]

2023-12-26 Thread via GitHub
Fokko commented on code in PR #132: URL: https://github.com/apache/iceberg-rust/pull/132#discussion_r1436335985 ## crates/iceberg/src/expr/mod.rs: ## @@ -0,0 +1,49 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See