Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364980016 ## crates/iceberg/src/avro/schema.rs: ## @@ -162,7 +168,7 @@ impl SchemaVisitor for SchemaToAvroSchema { let value_field = { let mut fi

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-18 Thread via GitHub
zhangminglei commented on code in PR #8844: URL: https://github.com/apache/iceberg/pull/8844#discussion_r1364968054 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -86,8 +86,8 @@ private CatalogProperties() {} * * Zero - Cache entries expires on

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364967043 ## crates/iceberg/src/avro/schema.rs: ## @@ -144,7 +150,7 @@ impl SchemaVisitor for SchemaToAvroSchema { // not string type. let key_fi

[PR] Add missing license headers [iceberg]

2023-10-18 Thread via GitHub
nastra opened a new pull request, #8875: URL: https://github.com/apache/iceberg/pull/8875 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364930268 ## crates/iceberg/src/avro/schema.rs: ## @@ -144,7 +150,7 @@ impl SchemaVisitor for SchemaToAvroSchema { // not string type. let key_field =

Re: [PR] fix(manifests): Make `added_snapshot_id` mandatory in V1 [iceberg-go]

2023-10-18 Thread via GitHub
nastra merged PR #23: URL: https://github.com/apache/iceberg-go/pull/23 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-18 Thread via GitHub
lirui-apache commented on code in PR #8844: URL: https://github.com/apache/iceberg/pull/8844#discussion_r1364866417 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -86,8 +86,8 @@ private CatalogProperties() {} * * Zero - Cache entries expires on

Re: [I] Flink: INSERT INTO worked with tuples but NOT writing data with select statement [iceberg]

2023-10-18 Thread via GitHub
a8356555 commented on issue #8874: URL: https://github.com/apache/iceberg/issues/8874#issuecomment-1769877830 provide my catalog and table setting ``` stenv.execute_sql(f''' CREATE CATALOG `mycatalog` WITH ( 'type' = 'iceberg', 'catalog-impl' = 'org.

[I] Flink: INSERT INTO worked with tuples but NOT writing data with select statement [iceberg]

2023-10-18 Thread via GitHub
a8356555 opened a new issue, #8874: URL: https://github.com/apache/iceberg/issues/8874 ### Apache Iceberg version 1.4.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 I'm using pyflink to write data to s3 bucket in iceberg format.

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364844632 ## crates/iceberg/src/spec/values.rs: ## @@ -996,10 +1008,552 @@ mod timestamptz { } } +mod serde { +use std::collections::BTreeMap; + +use crate

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364843043 ## crates/iceberg/src/spec/values.rs: ## @@ -996,10 +1008,552 @@ mod timestamptz { } } +mod serde { +use std::collections::BTreeMap; + +use crate::{ +

Re: [PR] Flink: Read parquet BINARY column as String for expected [iceberg]

2023-10-18 Thread via GitHub
fengjiajie commented on PR #8808: URL: https://github.com/apache/iceberg/pull/8808#issuecomment-1769826927 > > * It seems that ORC is not experiencing this issue because it creates value reader based on the iceberg column types. > > * Avro reads the fields entirely based on the file type,

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364825386 ## crates/iceberg/src/spec/values.rs: ## @@ -996,10 +1008,552 @@ mod timestamptz { } } +mod serde { +use std::collections::BTreeMap; + +use crate

Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1364823837 ## crates/catalog/rest/src/catalog.rs: ## @@ -0,0 +1,845 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1364818198 ## crates/catalog/rest/src/catalog.rs: ## @@ -0,0 +1,845 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1364807441 ## crates/catalog/rest/src/catalog.rs: ## @@ -0,0 +1,845 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] Add sort_order_id to STATS_COLUMNS to address null sort order ID in p… [iceberg]

2023-10-18 Thread via GitHub
zhangminglei commented on PR #8873: URL: https://github.com/apache/iceberg/pull/8873#issuecomment-1769792915 @Fokko Could you take a look on this fix ? Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-18 Thread via GitHub
zhangminglei commented on PR #8844: URL: https://github.com/apache/iceberg/pull/8844#issuecomment-1769792110 Hi, @nastra @lirui-apache I think all the PR comments have been addressed now - can you please check if anything else is required, and if not, merge this in? Thanks a lot! -- This

[PR] Add sort_order_id to STATS_COLUMNS to address null sort order ID in p… [iceberg]

2023-10-18 Thread via GitHub
zhangminglei opened a new pull request, #8873: URL: https://github.com/apache/iceberg/pull/8873 …lanned data files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364796056 ## crates/iceberg/src/spec/values.rs: ## @@ -996,10 +1008,552 @@ mod timestamptz { } } +mod serde { +use std::collections::BTreeMap; + +use crate::{ +

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
liurenjie1024 commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1364771184 ## crates/iceberg/src/avro/schema.rs: ## @@ -162,7 +168,7 @@ impl SchemaVisitor for SchemaToAvroSchema { let value_field = { let

Re: [PR] support read meta columns in staged scan [iceberg]

2023-10-18 Thread via GitHub
zinking commented on PR #8872: URL: https://github.com/apache/iceberg/pull/8872#issuecomment-1769779306 link the issue https://github.com/apache/iceberg/issues/8866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[PR] support read meta columns in staged scan [iceberg]

2023-10-18 Thread via GitHub
zinking opened a new pull request, #8872: URL: https://github.com/apache/iceberg/pull/8872 like in [query file meta column](https://docs.databricks.com/en/ingestion/file-metadata-column.html), it is convenient to support the following ``` spark.read \ .format("csv"

Re: [PR] Core: Reduce unnecessary add operations in deletedPaths set [iceberg]

2023-10-18 Thread via GitHub
bknbkn commented on PR #8868: URL: https://github.com/apache/iceberg/pull/8868#issuecomment-1769775332 cc @RussellSpitzer. @rdblue, may I request your assistance in enabling workflows approval? Thanks for your time. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364768700 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,21 +182,76 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [I] Failure to delete with merge-on-read in branch and main branch [iceberg]

2023-10-18 Thread via GitHub
edgarRd commented on issue #7635: URL: https://github.com/apache/iceberg/issues/7635#issuecomment-1769711711 Ping @amogh-jahagirdar - Is there any progress on this? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Writers object of PartitionedDeltaWriter will cause OOM when partition number is big [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] commented on issue #7216: URL: https://github.com/apache/iceberg/issues/7216#issuecomment-1769668109 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Writers object of PartitionedDeltaWriter will cause OOM when partition number is big [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] closed issue #7216: Writers object of PartitionedDeltaWriter will cause OOM when partition number is big URL: https://github.com/apache/iceberg/issues/7216 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Is flink support upsert data in iceberg table in streaming mode? [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] commented on issue #7220: URL: https://github.com/apache/iceberg/issues/7220#issuecomment-1769668086 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Is flink support upsert data in iceberg table in streaming mode? [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] closed issue #7220: Is flink support upsert data in iceberg table in streaming mode? URL: https://github.com/apache/iceberg/issues/7220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Wap branch does not support reading from the partitions table [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] closed issue #7297: Wap branch does not support reading from the partitions table URL: https://github.com/apache/iceberg/issues/7297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Wap branch does not support reading from the partitions table [iceberg]

2023-10-18 Thread via GitHub
github-actions[bot] commented on issue #7297: URL: https://github.com/apache/iceberg/issues/7297#issuecomment-1769668052 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] DeleteOrphanFiles or ExpireSnapshots outofmemory [iceberg]

2023-10-18 Thread via GitHub
RLashofRegas commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-1769511550 Ah. Thanks! setting `spark.sql.autoBroadcastJoinThreshold=-1` fixed it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Query fails when executed without filter i.e. aggregate pushdown [iceberg]

2023-10-18 Thread via GitHub
paulpaul1076 commented on issue #8859: URL: https://github.com/apache/iceberg/issues/8859#issuecomment-1769491751 @atifiu have you tried giving your application more memory? How much memory are you specifying in spark submit? -- This is an automated message from the Apache Git Service. To

Re: [I] DeleteOrphanFiles or ExpireSnapshots outofmemory [iceberg]

2023-10-18 Thread via GitHub
RussellSpitzer commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-1769486606 That's the table metadata. If your JSON file is 8gb large you have other problems :) -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] DeleteOrphanFiles or ExpireSnapshots outofmemory [iceberg]

2023-10-18 Thread via GitHub
RLashofRegas commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-1769378796 @RussellSpitzer Maybe I'm missing something but I don't think I can disable the broadcast in this case. `spark.sql.autoBroadcastJoinThreshold=-1` will only disable automatic bro

Re: [PR] AWS, Core: Chore replacing null/empty checks with Apache Commons StringUtils [iceberg]

2023-10-18 Thread via GitHub
amogh-jahagirdar closed pull request #8871: AWS, Core: Chore replacing null/empty checks with Apache Commons StringUtils URL: https://github.com/apache/iceberg/pull/8871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] AWS, Core: Chore replacing null/empty checks with Apache Commons StringUtils [iceberg]

2023-10-18 Thread via GitHub
amogh-jahagirdar commented on PR #8871: URL: https://github.com/apache/iceberg/pull/8871#issuecomment-1769370005 actually nvm, I think this does go against established practices -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]

2023-10-18 Thread via GitHub
haizhou-zhao commented on issue #8869: URL: https://github.com/apache/iceberg/issues/8869#issuecomment-1769293387 @syun64 In my org, we have very similar situation where we, unfortunately, can only use an internal procedure to grab auth token (that is quite different from OIDC flow). Based

Re: [PR] AWS: Glue catalog strip trailing slash on DB URI [iceberg]

2023-10-18 Thread via GitHub
dramaticlly commented on code in PR #8870: URL: https://github.com/apache/iceberg/pull/8870#discussion_r1364512879 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java: ## @@ -281,6 +281,7 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier

Re: [PR] AWS: Glue catalog strip trailing slash on DB URI [iceberg]

2023-10-18 Thread via GitHub
dramaticlly commented on code in PR #8870: URL: https://github.com/apache/iceberg/pull/8870#discussion_r1364512879 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java: ## @@ -281,6 +281,7 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364490041 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,21 +182,76 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364483076 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,65 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] fix(manifests): Make `added_snapshot_id` mandatory in V1 [iceberg-go]

2023-10-18 Thread via GitHub
zeroshade commented on PR #23: URL: https://github.com/apache/iceberg-go/pull/23#issuecomment-1769227993 CC @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[PR] fix(manifests): Make `added_snapshot_id` mandatory in V1 [iceberg-go]

2023-10-18 Thread via GitHub
zeroshade opened a new pull request, #23: URL: https://github.com/apache/iceberg-go/pull/23 Updating that `added_snapshot_id` is mandatory in both V1 and V2 as per https://github.com/apache/iceberg-go/pull/3#discussion_r1354308095 -- This is an automated message from the Apache Git Servic

Re: [I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]

2023-10-18 Thread via GitHub
syun64 commented on issue #8869: URL: https://github.com/apache/iceberg/issues/8869#issuecomment-1769215467 @danielcweeks and @Fokko I've raised this issue to continue our discussion on the previously closed PyIceberg PR on this parent repository. I've done some follow up research r

Re: [I] Distributed execution of DeleteReachableFilesSparkAction [iceberg]

2023-10-18 Thread via GitHub
RussellSpitzer commented on issue #8862: URL: https://github.com/apache/iceberg/issues/8862#issuecomment-1769209631 Yep, in our experience we have also seen a single driver is sufficient to issue a very large number of deletes. -- This is an automated message from the Apache Git Service.

Re: [PR] AWS: Glue catalog strip trailing slash on DB URI [iceberg]

2023-10-18 Thread via GitHub
amogh-jahagirdar commented on PR #8870: URL: https://github.com/apache/iceberg/pull/8870#issuecomment-1769112959 cc @okayhooni I just took another look at the issue and see that the DB URI `s3://test_bucket/temp_beta` does *not* have a trailing slash, so maybe my fix is insufficient. Can y

[I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]

2023-10-18 Thread via GitHub
syun64 opened a new issue, #8869: URL: https://github.com/apache/iceberg/issues/8869 ### Feature Request / Improvement The design of OAuth2 enables [Separation of Roles](https://www.oauth.com/oauth2-servers/differences-between-oauth-1-2/separation-of-roles/) between the API server an

Re: [I] Query fails when executed without filter i.e. aggregate pushdown [iceberg]

2023-10-18 Thread via GitHub
atifiu commented on issue #8859: URL: https://github.com/apache/iceberg/issues/8859#issuecomment-1769001351 @huaxingao I am unable to understand why OOM when not filtering the data but when filtering the data with selecting all the partitions without skipping anything works fine. Ideally in

Re: [I] Query fails when executed without filter i.e. aggregate pushdown [iceberg]

2023-10-18 Thread via GitHub
huaxingao commented on issue #8859: URL: https://github.com/apache/iceberg/issues/8859#issuecomment-1768982745 Seems it caused by OOM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364202095 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,65 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-18 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1364171669 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -174,8 +176,9 @@ public PartitionData copy() { * * @param toCopy a generic data file to copy.

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
dimas-b commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364200900 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#issuecomment-1768931859 I have send a new PR #82 for value part. Let's complete it first and then continue on working this PR later. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364196789 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#issuecomment-1768932760 cc @JanKaul @Fokko @Xuanwo @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] feat: support ser/deser of value [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME opened a new pull request, #82: URL: https://github.com/apache/iceberg-rust/pull/82 This PR: 1. modify to store i128 for Value::Decimal It make more easy to process Value::Decimal and avoid type consisent problem if we use Decimal. 2. support support ser/deser of value 3

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364159020 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +144,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364156037 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -225,8 +243,9 @@ public void createViewErrorCases() { .withQuery(trino.di

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364145699 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +137,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364135298 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +144,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
dimas-b commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364134288 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,65 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-18 Thread via GitHub
stevenzwu commented on PR #8803: URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1768776022 > > @pvary I think we probably want to push the `copyStatsForColumns` down to ManifestReader. https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestR

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
rdblue commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364092801 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +144,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
rdblue commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364091288 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -225,8 +243,9 @@ public void createViewErrorCases() { .withQuery(trino.di

Re: [PR] Remove set-out command (deprecated) and use GITHUB_OUTPUT instead [iceberg]

2023-10-18 Thread via GitHub
jbonofre closed pull request #8867: Remove set-out command (deprecated) and use GITHUB_OUTPUT instead URL: https://github.com/apache/iceberg/pull/8867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Remove set-out command (deprecated) and use GITHUB_OUTPUT instead [iceberg]

2023-10-18 Thread via GitHub
jbonofre commented on PR #8867: URL: https://github.com/apache/iceberg/pull/8867#issuecomment-1768741516 @nastra good catch ! I missed this one. I close this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
rdblue commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364089610 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -1446,7 +1544,14 @@ public void updateViewLocationConflict() { // the view was already

Re: [PR] Remove set-out command (deprecated) and use GITHUB_OUTPUT instead [iceberg]

2023-10-18 Thread via GitHub
nastra commented on PR #8867: URL: https://github.com/apache/iceberg/pull/8867#issuecomment-1768737358 looks like https://github.com/apache/iceberg/pull/8666 was opened recently that fixed the exact same thing. @jbonofre could you please review https://github.com/apache/iceberg/pull/8666 an

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364085322 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,65 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
rdblue commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1364084512 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +137,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Open-API: Make error required [iceberg]

2023-10-18 Thread via GitHub
nastra merged PR #8765: URL: https://github.com/apache/iceberg/pull/8765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364072720 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1364073343 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,65 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [I] Query fails when executed without filter i.e. aggregate pushdown [iceberg]

2023-10-18 Thread via GitHub
atifiu commented on issue #8859: URL: https://github.com/apache/iceberg/issues/8859#issuecomment-1768659309 @paulpaul1076 I have now pasted the complete error stack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Query fails when executed without filter i.e. aggregate pushdown [iceberg]

2023-10-18 Thread via GitHub
paulpaul1076 commented on issue #8859: URL: https://github.com/apache/iceberg/issues/8859#issuecomment-1768545557 There should be a wrapped exception and explains what exactly happened in the parallel task. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
ZENOTME commented on PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#issuecomment-1768546364 I will split the RawLiteral out into another PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363956106 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1363931200 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -56,10 +59,16 @@ public abstract class ViewCatalogTests

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1363926224 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -664,6 +684,41 @@ public void renameViewTargetAlreadyExistsAsTable() { .hasMessageCon

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-18 Thread via GitHub
nastra commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1363922938 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -56,10 +59,16 @@ public abstract class ViewCatalogTests

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363889171 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363886834 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363874134 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +181,38 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [PR] Nessie: retain authorship information when creating and dropping namespaces [iceberg]

2023-10-18 Thread via GitHub
ajantha-bhat commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363865522 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) {

Re: [PR] Encryption integration and test [iceberg]

2023-10-18 Thread via GitHub
ggershinsky commented on PR #5544: URL: https://github.com/apache/iceberg/pull/5544#issuecomment-1768384984 The current PR sequence develops the encryption tools using Spark as a usecase. But once the toolset converges, we'll certainly need the Flink support and unitests. -- This is an a

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363824086 ## crates/iceberg/src/spec/values.rs: ## @@ -966,6 +978,547 @@ mod timestamptz { } } +mod serde { +use std::collections::{BTreeMap, HashMap}; + +use crat

Re: [PR] Nessie: retain authorship information when creating a namespace [iceberg]

2023-10-18 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1363786870 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,23 +183,52 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363761173 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,671 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [I] How to release [iceberg-rust]

2023-10-18 Thread via GitHub
Xuanwo commented on issue #81: URL: https://github.com/apache/iceberg-rust/issues/81#issuecomment-1768322591 Sure, let me do this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363758398 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,671 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-18 Thread via GitHub
barronw commented on code in PR #76: URL: https://github.com/apache/iceberg-rust/pull/76#discussion_r1363755844 ## crates/iceberg/src/avro/schema.rs: ## @@ -96,7 +98,13 @@ impl SchemaVisitor for SchemaToAvroSchema { _struct: &StructType, results: Vec, ) ->

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-18 Thread via GitHub
barronw commented on PR #76: URL: https://github.com/apache/iceberg-rust/pull/76#issuecomment-1768315223 > @barronw This is great! One question, do we need the Avro file (the one in `testdata/`)? I don't see it being referenced, and we generally try to avoid binary data in the repository.

[PR] Remove set-out command (deprecated) and use GITHUB_OUTPUT instead [iceberg]

2023-10-18 Thread via GitHub
jbonofre opened a new pull request, #8867: URL: https://github.com/apache/iceberg/pull/8867 set-out command is now deprecated in GitHub Actions. Instead we should use GITHUB_OUTPUT env variable. See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-se

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363751192 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,671 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363750143 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,671 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-10-18 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1363748144 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,671 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

[I] support meta column query on staged scan [iceberg]

2023-10-18 Thread via GitHub
zinking opened a new issue, #8866: URL: https://github.com/apache/iceberg/issues/8866 ### Apache Iceberg version 1.4.0 (latest release) ### Query engine None ### Please describe the bug 🐞 ``` Dataset scanDF = spark .read()

Re: [I] SortOrderId is null after planFile operation [iceberg]

2023-10-18 Thread via GitHub
zhangminglei commented on issue #8864: URL: https://github.com/apache/iceberg/issues/8864#issuecomment-1768304382 Yes. I am interested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

  1   2   >