Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1528524348 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -554,10 +556,6 @@ private Map convertToMetadata(Database database) { } Database

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528543356 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528554934 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528558870 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528560412 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528560412 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528562258 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528564037 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528568136 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528590383 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

[I] Improvements for manifest file caching [iceberg]

2024-03-18 Thread via GitHub
tdcmeehan opened a new issue, #9991: URL: https://github.com/apache/iceberg/issues/9991 ### Feature Request / Improvement While experimenting with the features in #4518 (core: Provide mechanism to cache manifest file content), we encountered a couple of limitations which we would lik

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528603864 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2991,76 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1528623059 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2991,76 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [PR] docs: Add links checker [iceberg]

2024-03-18 Thread via GitHub
manuzhang commented on code in PR #9965: URL: https://github.com/apache/iceberg/pull/9965#discussion_r1528624029 ## .github/workflows/docs-check-links.yml: ## @@ -0,0 +1,39 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Fix `TestStrictMetricsEvaluator` assertion message [iceberg]

2024-03-18 Thread via GitHub
Fokko closed pull request #9938: Fix `TestStrictMetricsEvaluator` assertion message URL: https://github.com/apache/iceberg/pull/9938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[PR] Fix `TestStrictMetricsEvaluator` assertion message [iceberg]

2024-03-18 Thread via GitHub
Fokko opened a new pull request, #9992: URL: https://github.com/apache/iceberg/pull/9992 All values (`5` and `6`) are not between the upper and lower bound of `[30, 79]`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] UUID literal to binary and fixed [iceberg-python]

2024-03-18 Thread via GitHub
sebpretzer opened a new pull request, #529: URL: https://github.com/apache/iceberg-python/pull/529 Closes https://github.com/apache/iceberg-python/issues/522 Notes: 1. `asserts isinstance(...` passes the tests but fails mypy. I didn't want to dig too far into this issue since I bel

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1528671767 ## crates/iceberg/src/scan.rs: ## @@ -159,6 +190,21 @@ impl TableScan { while let Some(entry) = entries.next().await { let manif

Re: [PR] UUID literal to binary and fixed [iceberg-python]

2024-03-18 Thread via GitHub
sebpretzer commented on PR #529: URL: https://github.com/apache/iceberg-python/pull/529#issuecomment-2004072143 @Fokko this is ready for your review whenever you have a chance, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Iceberg Rest Catalog Support for a Separate OIDC Authorization Server URI [iceberg]

2024-03-18 Thread via GitHub
syun64 closed issue #8869: Iceberg Rest Catalog Support for a Separate OIDC Authorization Server URI URL: https://github.com/apache/iceberg/issues/8869 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] [WIP] Migrate Files, Append and Overwrite files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
tomtongue opened a new pull request, #9994: URL: https://github.com/apache/iceberg/pull/9994 Migrate the following test classes in iceberg-core to JUnit 5 and AssertJ style for https://github.com/apache/iceberg/issues/9085. ## Current Progress - [x] `TestFastAppend.java` - [ ]

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
ZENOTME commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2004158618 Thanks for this job! > I'm quite unsure about the fn transform and the handling of the arrow_array - seems kinda clunky? I think the transform can provide an interface li

[PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon opened a new pull request, #281: URL: https://github.com/apache/iceberg-rust/pull/281 The purpose of the PR is to change the `ManifestEntry` properties and `FileScanTask` `data_file()` method to public in order to enable public access to the file metadata API. Some consumers are

[I] Making FileScanTask data_file APi public [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon opened a new issue, #282: URL: https://github.com/apache/iceberg-rust/issues/282 According to the original design, `FileScanTask` can be transformed to arrow record batch stream for query purposes, however, there are cases in which consumers or API users will be interested in access

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-18 Thread via GitHub
ZENOTME commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1528775973 ## crates/iceberg/src/scan.rs: ## @@ -214,11 +273,236 @@ impl FileScanTask { } } +/// Evaluates manifest files to see if their partition values comply with a

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-18 Thread via GitHub
dimas-b commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1528789710 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieCatalog.java: ## @@ -169,4 +171,25 @@ public void testWarehouseLocationWithTrailingSlash() {

Re: [PR] Fix `TestStrictMetricsEvaluator` assertion message [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9992: URL: https://github.com/apache/iceberg/pull/9992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-18 Thread via GitHub
ZENOTME commented on code in PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#discussion_r1528887679 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -727,14 +746,10 @@ pub(super) mod _serde { .collect(), default_spec_id: v.d

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-18 Thread via GitHub
ZENOTME commented on code in PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#discussion_r1528887679 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -727,14 +746,10 @@ pub(super) mod _serde { .collect(), default_spec_id: v.d

[PR] Glue endpoint config variable [iceberg-python]

2024-03-18 Thread via GitHub
sebpretzer opened a new pull request, #530: URL: https://github.com/apache/iceberg-python/pull/530 Closes: https://github.com/apache/iceberg-python/issues/414 I don't know java, but tried to stay true to [the implementation](https://github.com/apache/iceberg/blob/053d54172fd903be9eda7

Re: [I] Support Nessie catalog [iceberg-python]

2024-03-18 Thread via GitHub
fraibacas commented on issue #19: URL: https://github.com/apache/iceberg-python/issues/19#issuecomment-2004518219 @ajantha-bhat Any rough idea about when this will be available? thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
viirya commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529000272 ## Cargo.toml: ## @@ -61,7 +61,7 @@ murmur3 = "0.5.2" once_cell = "1" opendal = "0.45" ordered-float = "4.0.0" -parquet = "50" +parquet = "51" Review Comment:

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
viirya commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529000892 ## crates/iceberg/src/transform/temporal.rs: ## @@ -17,10 +17,8 @@ use super::TransformFunction; use crate::{Error, ErrorKind, Result}; -use arrow_arith::{ -a

Re: [I] fix: Making FileScanTask data_file APi public [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon commented on issue #282: URL: https://github.com/apache/iceberg-rust/issues/282#issuecomment-2004577238 WIP on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Table commit retries based on table properties [iceberg-python]

2024-03-18 Thread via GitHub
Buktoria commented on PR #330: URL: https://github.com/apache/iceberg-python/pull/330#issuecomment-2004665217 So I made a large fundamental change to the original design, where catalogs need to implement a function where they declare what exceptions are retryable. This becomes the bridge be

[PR] `add_files` support partitioned tables [iceberg-python]

2024-03-18 Thread via GitHub
syun64 opened a new pull request, #531: URL: https://github.com/apache/iceberg-python/pull/531 As a follow up to https://github.com/apache/iceberg-python/pull/506, this PR introduces the support for adding files as DataFiles to partitioned tables. Instead of relying on the more inaccu

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-18 Thread via GitHub
nk1506 commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1529067654 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,25 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-18 Thread via GitHub
nk1506 commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1529069107 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -554,10 +556,6 @@ private Map convertToMetadata(Database database) { } Database

Re: [PR] `add_files` support partitioned tables [iceberg-python]

2024-03-18 Thread via GitHub
syun64 commented on PR #531: URL: https://github.com/apache/iceberg-python/pull/531#issuecomment-2004681281 If the implementation looks good, I can make the corresponding updates to the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[PR] Add as_arrow() to Schema class [iceberg-python]

2024-03-18 Thread via GitHub
ndrluis opened a new pull request, #532: URL: https://github.com/apache/iceberg-python/pull/532 Use case ``` import pyarrow as pa import pandas as pd table = catalog.load_table('some_schema.table_name') schema = table.schema().as_arrow() data = { 'id_job':

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon commented on PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#issuecomment-2004766897 Thanks for fixing this. I was also struggling with understanding why it fails -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Add as_arrow() to Schema class [iceberg-python]

2024-03-18 Thread via GitHub
Fokko commented on code in PR #532: URL: https://github.com/apache/iceberg-python/pull/532#discussion_r1529234952 ## pyiceberg/schema.py: ## @@ -180,6 +182,12 @@ def as_struct(self) -> StructType: """Return the schema as a struct.""" return StructType(*self.fie

[I] Snapshot sets `OVERWRITE` while `DELETE` is expected [iceberg]

2024-03-18 Thread via GitHub
Fokko opened a new issue, #9995: URL: https://github.com/apache/iceberg/issues/9995 ### Feature Request / Improvement ![image](https://github.com/apache/iceberg/assets/1134248/d4e523dc-cbd5-4cf5-b4c1-c07672f8969a) ### Query engine None -- This is an automated mess

[PR] core: Filter on live entries when reading the manifest [iceberg]

2024-03-18 Thread via GitHub
Fokko opened a new pull request, #9996: URL: https://github.com/apache/iceberg/pull/9996 This will reduce the allocation of objects and filter out irrelevant manifests at read time. PS: In the diff, the indentation looks a bit off. -- This is an automated message from the Apache Gi

Re: [PR] Add as_arrow() to Schema class [iceberg-python]

2024-03-18 Thread via GitHub
kevinjqliu commented on PR #532: URL: https://github.com/apache/iceberg-python/pull/532#issuecomment-2005060452 (reposting from slack) This is great. Exposing the Iceberg Table's schema as pyarrow schema will also solve the problem I was having here https://github.com/apache/iceb

Re: [PR] UUID literal to binary and fixed [iceberg-python]

2024-03-18 Thread via GitHub
Fokko merged PR #529: URL: https://github.com/apache/iceberg-python/pull/529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Cast from UUIDLiteral to other types? [iceberg-python]

2024-03-18 Thread via GitHub
Fokko closed issue #522: Cast from UUIDLiteral to other types? URL: https://github.com/apache/iceberg-python/issues/522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Glue endpoint config variable [iceberg-python]

2024-03-18 Thread via GitHub
Fokko commented on code in PR #530: URL: https://github.com/apache/iceberg-python/pull/530#discussion_r1529322025 ## pyiceberg/catalog/glue.py: ## @@ -105,6 +105,10 @@ GLUE_SKIP_ARCHIVE = "glue.skip-archive" GLUE_SKIP_ARCHIVE_DEFAULT = True +# Configure an alternative endpoi

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
hussein-awala commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-2005106161 > In that case I would probably just add a new test class where reading and writing is done through Spark. I think the purpose of TestSparkReaderWithBloomFilter was to actually only

Re: [I] Caused by: java.net.SocketException: Connection reset [iceberg]

2024-03-18 Thread via GitHub
javrasya commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-2005133969 The code I share earlier @shanzi caused some data loss. It was not closing the currently open stream, so be careful with that. I am sorry see that my buggy code above is spreading.

Re: [I] Caused by: java.net.SocketException: Connection reset [iceberg]

2024-03-18 Thread via GitHub
javrasya commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-2005144637 I have a strong feeling that the Timeout error you get is also due to my buggy code. Just because it does not close the stream when socket closed exception is received, it ends up g

Re: [PR] Migrate Files, Append and Overwrite files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
tomtongue commented on PR #9994: URL: https://github.com/apache/iceberg/pull/9994#issuecomment-2005245722 @nastra Change the classes in Core to JUnit5. Could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Add metadata tables [iceberg-python]

2024-03-18 Thread via GitHub
ndrluis commented on issue #511: URL: https://github.com/apache/iceberg-python/issues/511#issuecomment-2005282586 @Fokko, could you assign the History table to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Spark: support use-table-distribution-and-ordering in session conf [iceberg]

2024-03-18 Thread via GitHub
javrasya commented on PR #8164: URL: https://github.com/apache/iceberg/pull/8164#issuecomment-2005393066 Any update on this. Due to the fact that Spark3.3 is the latest version supported by Glue, we are stuck with it and not being able to set this within SQL is unreasonable since it will so

Re: [I] Cannot write incompatible dataset to table with schema error for list types [iceberg]

2024-03-18 Thread via GitHub
github-actions[bot] commented on issue #2192: URL: https://github.com/apache/iceberg/issues/2192#issuecomment-2005403954 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Hive: got error while joining iceberg table and hive table [iceberg]

2024-03-18 Thread via GitHub
github-actions[bot] commented on issue #2198: URL: https://github.com/apache/iceberg/issues/2198#issuecomment-2005404059 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Need help inserting data into hadoop table with flink sql in java [iceberg]

2024-03-18 Thread via GitHub
github-actions[bot] commented on issue #2209: URL: https://github.com/apache/iceberg/issues/2209#issuecomment-2005404175 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[I] [core] Table metadata points to metadata.json that doesn't exist as it failed to upload [iceberg]

2024-03-18 Thread via GitHub
abmo-x opened a new issue, #9997: URL: https://github.com/apache/iceberg/issues/9997 ### Apache Iceberg version 1.3.1 ### Query engine Flink ### Please describe the bug 🐞 Iceberg table points to a metadata.json that failed to upload to s3. Due to this the t

[PR] [core] fix #9997 - throw io exception on close if S3a put object interrupted [iceberg]

2024-03-18 Thread via GitHub
abmo-x opened a new pull request, #9998: URL: https://github.com/apache/iceberg/pull/9998 This is fix for https://github.com/apache/iceberg/issues/9997 ### Root cause s3a putObject was interrupted due to flink pipeline failure. As this interrupt is not handled and thrown as an

Re: [PR] [core] fix #9997 - throw io exception on close if S3a put object interrupted [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on PR #9998: URL: https://github.com/apache/iceberg/pull/9998#issuecomment-2005537501 @rdblue I reproduced and verified this locally with an Integration test. I am wondering whats the best way to include a test for this. I didn't see any tests using hadoop-aws mo

Re: [PR] [core] fix #9997 - throw io exception on close if S3a put object interrupted [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529549780 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1529564049 ## crates/iceberg/src/expr/predicate.rs: ## @@ -282,15 +282,18 @@ impl Bind for Predicate { return Ok(BoundPredicate::AlwaysTrue);

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1529565534 ## crates/iceberg/src/expr/predicate.rs: ## @@ -282,15 +282,18 @@ impl Bind for Predicate { return Ok(BoundPredicate::AlwaysTrue);

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1529565772 ## crates/iceberg/src/expr/predicate.rs: ## Review Comment: Could we also add some ut for negates? -- This is an automated message from the Apache Git

Re: [PR] [core] fix #9997 - throw io exception on close if S3a put object interrupted [iceberg]

2024-03-18 Thread via GitHub
mas-chen commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529572040 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that don't exists [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529575056 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that don't exists [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529575056 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that don't exists [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529575056 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that don't exists [iceberg]

2024-03-18 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1529576284 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#discussion_r1529576423 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -297,19 +298,37 @@ impl TableMetadataBuilder { properties, } = table_creation; -

Re: [I] Implement `project` for `Transform`. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on issue #264: URL: https://github.com/apache/iceberg-rust/issues/264#issuecomment-2005612750 @marvinlanhenke Sorry for late reply. Yeah, this is exactly what I'm thinking about, thanks! -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-2005613132 @marvinlanhenke The pr looks great, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#discussion_r1529594580 ## crates/iceberg/src/expr/predicate.rs: ## @@ -187,6 +199,14 @@ impl SetExpression { debug_assert!(op.is_set()); Self { op, term, literals }

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2005621939 Hi, @marvinlanhenke This pr looks great to me. But I think @ZENOTME is right, we should implement #283 first. The reason we implement transform on arrow array could be found [he

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
ZENOTME commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2005643728 > Hi, @marvinlanhenke This pr looks great to me. But I think @ZENOTME is right, we should implement #283 first. The reason we implement transform on arrow array could be found [here](

Re: [I] Discussion: How to handle S3 Config in Catalog [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on issue #273: URL: https://github.com/apache/iceberg-rust/issues/273#issuecomment-2005686824 @marvinlanhenke Sorry for late reply. I think this approach is fine, and java/python also follows this approach. -- This is an automated message from the Apache Git Servic

Re: [I] Setup `TableMetadataBuilder`. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 closed issue #250: Setup `TableMetadataBuilder`. URL: https://github.com/apache/iceberg-rust/issues/250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] Setup `TableMetadataBuilder`. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on issue #250: URL: https://github.com/apache/iceberg-rust/issues/250#issuecomment-2005692494 Closed by #262 . Feel free to open when necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#issuecomment-2005697218 cc @Xuanwo PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529647136 ## Cargo.toml: ## @@ -61,7 +61,7 @@ murmur3 = "0.5.2" once_cell = "1" opendal = "0.45" ordered-float = "4.0.0" -parquet = "50" +parquet = "51" Review Comme

Re: [PR] fix: chrono dep [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on PR #274: URL: https://github.com/apache/iceberg-rust/pull/274#issuecomment-2005699197 Hi, @odysa Thanks for this pr. I think this could be solved by `cargo update`? As a library, it's usually not good practice to fix it to a minor version. -- This is an automat

Re: [PR] docs: annotate precision and length to primitive types [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 merged PR #270: URL: https://github.com/apache/iceberg-rust/pull/270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata including readonly flag [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on PR #267: URL: https://github.com/apache/iceberg-rust/pull/267#issuecomment-2005702012 cc @Xuanwo @Fokko PTAL Hi, @a-agmon Would you mind to resolve the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
Xuanwo commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529658003 ## Cargo.toml: ## @@ -61,7 +61,7 @@ murmur3 = "0.5.2" once_cell = "1" opendal = "0.45" ordered-float = "4.0.0" -parquet = "50" +parquet = "51" Review Comment:

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
Xuanwo commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529658358 ## Cargo.toml: ## @@ -61,7 +61,7 @@ murmur3 = "0.5.2" once_cell = "1" opendal = "0.45" ordered-float = "4.0.0" -parquet = "50" +parquet = "51" Review Comment:

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata including readonly flag [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon closed pull request #267: feat (static table): implement a read-only table struct loaded from metadata including readonly flag URL: https://github.com/apache/iceberg-rust/pull/267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata including readonly flag [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon commented on PR #267: URL: https://github.com/apache/iceberg-rust/pull/267#issuecomment-2005710613 That's a duplicate. Closing it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Iceberg does not work with other types of data lakes [iceberg]

2024-03-18 Thread via GitHub
BsoBird closed issue #8624: Iceberg does not work with other types of data lakes URL: https://github.com/apache/iceberg/issues/8624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
himadripal commented on PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#issuecomment-2005781437 Thanks for fixing this @viirya, I tried so many things, thinking its a problem in my local setup. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1529731668 ## Cargo.toml: ## @@ -61,7 +61,7 @@ murmur3 = "0.5.2" once_cell = "1" opendal = "0.45" ordered-float = "4.0.0" -parquet = "50" +parquet = "51" Review Comme

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 merged PR #280: URL: https://github.com/apache/iceberg-rust/pull/280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] CI looks like broken [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 closed issue #279: CI looks like broken URL: https://github.com/apache/iceberg-rust/issues/279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#issuecomment-2005809328 @liurenjie1024 Thanks for the review, I will apply those changes / suggestions. I was testing / using Trino as a client. -- This is an automated message from the Apa

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2005814209 > Hi, @marvinlanhenke This pr looks great to me. But I think @ZENOTME is right, we should implement #283 first. The reason we implement transform on arrow array could be found

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2005814739 > > Hi, @marvinlanhenke This pr looks great to me. But I think @ZENOTME is right, we should implement #283 first. The reason we implement transform on arrow array could be found

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
viirya commented on PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#issuecomment-2005905674 Thanks all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon opened a new pull request, #284: URL: https://github.com/apache/iceberg-rust/pull/284 resolves #282 The purpose of the PR is to change the ManifestEntry properties and FileScanTask data_file() method to public in order to enable public access to the file metadata API. Some co

Re: [I] Implement `project` for `Transform`. [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on issue #264: URL: https://github.com/apache/iceberg-rust/issues/264#issuecomment-2005921495 thank you, once #283 is ready, I will continue here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon closed pull request #281: fix: enable public access to ManifestEntry properties URL: https://github.com/apache/iceberg-rust/pull/281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-18 Thread via GitHub
a-agmon commented on PR #281: URL: https://github.com/apache/iceberg-rust/pull/281#issuecomment-2005923666 Closing this re-opening a new PR after fix to CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

<    1   2   3   >