Re: [PR] Spark: Fail on recursive cycle in view [iceberg]

2024-03-26 Thread via GitHub
amogh-jahagirdar merged PR #9834: URL: https://github.com/apache/iceberg/pull/9834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Non-nullable columns marked as nullable during table creation [iceberg]

2024-03-26 Thread via GitHub
vamsibokam commented on issue #7771: URL: https://github.com/apache/iceberg/issues/7771#issuecomment-2022070117 @zhongyujiang, Could you elaborate on how to preserve the nullability constraints using the above [change](https://github.com/apache/spark/pull/41070) mentioned above? Should we

Re: [I] Read Parquet data file with projection [iceberg-rust]

2024-03-26 Thread via GitHub
viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2022062466 > > I think this Implement sth like ArrowProjectionVisitor is for the following two items? > > Yes, but with one extra requirement: reconstructing struct arrays. For example

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1540493957 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +368,53 @@ public void renameTable(TableIdentifier from, TableIdentifier origina

Re: [PR] Migrate WAP, Metrics and FilterFiles in Core to JUnit5 [iceberg]

2024-03-26 Thread via GitHub
tomtongue commented on PR #10039: URL: https://github.com/apache/iceberg/pull/10039#issuecomment-2021930566 @nastra Additional classes are migrated. Could you review them? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 merged PR #276: URL: https://github.com/apache/iceberg-rust/pull/276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1540424178 ## crates/iceberg/src/expr/predicate.rs: ## Review Comment: Sorry, I didn't notice that before. -- This is an automated message from the Apache Git Se

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1537479886 ## crates/iceberg/src/arrow.rs: ## @@ -49,10 +54,17 @@ impl ArrowReaderBuilder { self } +/// Sets the desired column projection with a list

Re: [I] Read Parquet data file with projection [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2021856565 > I think this Implement sth like ArrowProjectionVisitor is for the following two items? Yes, but with one extra requirement: reconstructing struct arrays. For exampl

Re: [PR] feat: add transform_literal [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 merged PR #287: URL: https://github.com/apache/iceberg-rust/pull/287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: Glue Catalog - namespace operations (2/3) [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 merged PR #304: URL: https://github.com/apache/iceberg-rust/pull/304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-26 Thread via GitHub
QuakeWang commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1529573159 ## crates/iceberg/src/expr/predicate.rs: ## @@ -282,15 +282,18 @@ impl Bind for Predicate { return Ok(BoundPredicate::AlwaysTrue);

Re: [PR] Parquet: Implement column index filter and update row read path to support page skipping [iceberg]

2024-03-26 Thread via GitHub
iflytek-hmwang5 commented on PR #6967: URL: https://github.com/apache/iceberg/pull/6967#issuecomment-2021745840 > Um, I'm not sure, I think it depends on the community's priorities, and reviewers' bandwidth. You can refer to this PR, https://github.com/apache/iceberg/issues/193, Anton Ok

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540327433 ## crates/catalog/rest/src/catalog.rs: ## @@ -103,8 +104,7 @@ impl RestCatalogConfig { ]) } -fn try_create_rest_client(&self) -> Result { -

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
liurenjie1024 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540327945 ## crates/catalog/rest/src/catalog.rs: ## @@ -103,8 +104,7 @@ impl RestCatalogConfig { ]) } -fn try_create_rest_client(&self) -> Result { -

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-26 Thread via GitHub
manuzhang commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2021729316 > Can you remove the retry config and we'll go with the current approach? Done. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Flink : add watermark support for streaming read [iceberg]

2024-03-26 Thread via GitHub
github-actions[bot] commented on issue #2264: URL: https://github.com/apache/iceberg/issues/2264#issuecomment-2021690527 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink : add Computed Column support for iceberg [iceberg]

2024-03-26 Thread via GitHub
github-actions[bot] commented on issue #2262: URL: https://github.com/apache/iceberg/issues/2262#issuecomment-2021690501 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Catalogs Do Not Easily Support Full State Rollback [iceberg]

2024-03-26 Thread via GitHub
github-actions[bot] closed issue #1944: Catalogs Do Not Easily Support Full State Rollback URL: https://github.com/apache/iceberg/issues/1944 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Catalogs Do Not Easily Support Full State Rollback [iceberg]

2024-03-26 Thread via GitHub
github-actions[bot] commented on issue #1944: URL: https://github.com/apache/iceberg/issues/1944#issuecomment-2021690200 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540214185 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -196,15 +265,14 @@ protected List scanColumns() { */ public BatchScan baseTable

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
flyrain commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540253733 ## crates/catalog/rest/src/catalog.rs: ## @@ -103,8 +104,7 @@ impl RestCatalogConfig { ]) } -fn try_create_rest_client(&self) -> Result { -

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540229802 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: ## @@ -458,6 +460,7 @@ public int totalGroupCount() {

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540229802 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: ## @@ -458,6 +460,7 @@ public int totalGroupCount() {

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540218841 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -278,28 +344,38 @@ public void close() throws IOException { @Override

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540218841 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -278,28 +344,38 @@ public void close() throws IOException { @Override

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540218841 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -278,28 +344,38 @@ public void close() throws IOException { @Override

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540218841 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -278,28 +344,38 @@ public void close() throws IOException { @Override

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1540213729 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -141,25 +147,88 @@ private Schema calculateSchema() { } } + /** + * Handle co

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on PR #10020: URL: https://github.com/apache/iceberg/pull/10020#issuecomment-2021611486 Redid the approach, now I re-assign partition field ids. I fix the initial problems, it requires a bit of finesse. The broad picture: - When we need to read manifests, we

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540191330 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-26 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1540155731 ## core/src/test/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java: ## @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540153323 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

2024-03-26 Thread via GitHub
syun64 commented on issue #368: URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2021532558 > In order to implement this with snapshot properties I want my writer to do the following transactionally: > Fetch the current snapshot's dateranges property. Modify th

Re: [I] Make OAuth token server configurable [iceberg-rust]

2024-03-26 Thread via GitHub
flyrain closed issue #291: Make OAuth token server configurable URL: https://github.com/apache/iceberg-rust/issues/291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] feat: Make OAuth token server configurable [iceberg-rust]

2024-03-26 Thread via GitHub
flyrain merged PR #305: URL: https://github.com/apache/iceberg-rust/pull/305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
flyrain commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1540025945 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#issuecomment-2021248780 Note: CI check failed due to a known issue https://github.com/apache/iceberg-rust/issues/307, which has been fixed. Rerun build should success. -- This is an automated message fro

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539932982 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +368,53 @@ public void renameTable(TableIdentifier from, TableIdentifier orig

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1539932487 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1539932487 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539907379 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -309,65 +304,20 @@ protected enum CommitStatus { * @return Commit Status of Succ

Re: [I] Avoid installing yanked chrono version v0.4.36 [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on issue #307: URL: https://github.com/apache/iceberg-rust/issues/307#issuecomment-2021214483 `https://crates.io/crates/chrono/0.4.36` has been marked as yanked 2 hours ago. Verified locally after `cargo update`. Closing the issue. -- This is an automated message from

Re: [I] Avoid installing yanked chrono version v0.4.36 [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 closed issue #307: Avoid installing yanked chrono version v0.4.36 URL: https://github.com/apache/iceberg-rust/issues/307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] Pin chrono version to avoid yanked version v0.4.36 [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 opened a new issue, #307: URL: https://github.com/apache/iceberg-rust/issues/307 The latest version of crates `chrono ` (just released today) is marked as yanked Today we set `chrono = "0.4"` in Cargo.toml, which end up install the late version v0.4.36 which has been marked

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2021163385 Thanks for the reviews, @rdblue and @Fokko! Please have another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539860114 ## api/src/test/java/org/apache/iceberg/expressions/TestStringLiteralConversions.java: ## @@ -132,6 +132,38 @@ public void testStringToTimestampLiteral() { .isEq

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539858300 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -107,17 +107,65 @@ public void testSpecValues() { .isEqualTo(-662762989); Lit

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539857850 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -298,7 +299,7 @@ public Literal to(Type type) { case TIME: return (Literal)

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539854973 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -501,19 +538,17 @@ public Literal to(Type type) { return (Literal) new TimeLiteral(ti

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539854708 ## api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java: ## @@ -120,19 +176,32 @@ public static long isoTimeToMicros(String timeString) { } public static l

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539854388 ## api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java: ## @@ -120,19 +176,32 @@ public static long isoTimeToMicros(String timeString) { } public static l

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539852804 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -426,23 +427,55 @@ protected Type.TypeID typeId() { } static class TimestampLiteral exten

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539852462 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -595,11 +613,17 @@ private static String sanitizeString(CharSequence value, long now, int

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539851397 ## api/src/main/java/org/apache/iceberg/transforms/Transforms.java: ## @@ -86,8 +87,9 @@ private Transforms() {} try { if (type.typeId() == Type.TypeID.TIME

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539850776 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -28,57 +29,131 @@ import org.apache.iceberg.relocated.com.google.common.base.Preconditions; im

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-03-26 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1539850016 ## api/src/main/java/org/apache/iceberg/transforms/Days.java: ## @@ -55,14 +56,15 @@ public boolean satisfiesOrderOf(Transform other) { } if (other instanceof

Re: [PR] feat: Support customized header in Rest catalog client [iceberg-rust]

2024-03-26 Thread via GitHub
flyrain commented on code in PR #306: URL: https://github.com/apache/iceberg-rust/pull/306#discussion_r1539833388 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +983,68 @@ mod tests { ); } +#[tokio::test] +async fn test_get_default_headers() { +

Re: [I] Cannot create table if location/endpoint is s3 on a "secure" Minio server [iceberg-python]

2024-03-26 Thread via GitHub
thinkORo closed issue #540: Cannot create table if location/endpoint is s3 on a "secure" Minio server URL: https://github.com/apache/iceberg-python/issues/540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Make AzureProperties w/ shared-key creds serializable [iceberg]

2024-03-26 Thread via GitHub
snazy opened a new pull request, #10045: URL: https://github.com/apache/iceberg/pull/10045 As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Migrate WAP, Metrics and FilterFiles in Core to JUnit5 [iceberg]

2024-03-26 Thread via GitHub
tomtongue commented on code in PR #10039: URL: https://github.com/apache/iceberg/pull/10039#discussion_r1539799719 ## core/src/test/java/org/apache/iceberg/TestMetrics.java: ## @@ -766,24 +771,38 @@ protected void assertCounts(int fieldId, Long valueCount, Long nullValueCount,

Re: [PR] Migrate WAP, Metrics and FilterFiles in Core to JUnit5 [iceberg]

2024-03-26 Thread via GitHub
tomtongue commented on code in PR #10039: URL: https://github.com/apache/iceberg/pull/10039#discussion_r1539796974 ## core/src/test/java/org/apache/iceberg/TestMetrics.java: ## @@ -766,24 +771,38 @@ protected void assertCounts(int fieldId, Long valueCount, Long nullValueCount,

Re: [I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

2024-03-26 Thread via GitHub
brianfromoregon commented on issue #368: URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2020990035 Hi @syun64, thanks for chiming in! My batch app store historical data, there is always a date column. It runs for each date and will insert data for that date. So

Re: [PR] Migrate WAP, Metrics and FilterFiles in Core to JUnit5 [iceberg]

2024-03-26 Thread via GitHub
tomtongue commented on code in PR #10039: URL: https://github.com/apache/iceberg/pull/10039#discussion_r1539738138 ## core/src/test/java/org/apache/iceberg/TestMetrics.java: ## @@ -766,24 +771,38 @@ protected void assertCounts(int fieldId, Long valueCount, Long nullValueCount,

Re: [PR] detect breaking changes [iceberg-python]

2024-03-26 Thread via GitHub
syun64 commented on PR #394: URL: https://github.com/apache/iceberg-python/pull/394#issuecomment-2020988016 @jaychia 'suggestion -> reorganize modules to the top level -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] feat: Make OAuth token server configurable [iceberg-rust]

2024-03-26 Thread via GitHub
whynick1 commented on code in PR #305: URL: https://github.com/apache/iceberg-rust/pull/305#discussion_r1539735723 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +964,39 @@ mod tests { ); } +#[tokio::test] +async fn test_oauth_with_auth_url() { +

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-03-26 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1539731305 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-03-26 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1539730660 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

Re: [I] Read Parquet data file with projection [iceberg-rust]

2024-03-26 Thread via GitHub
viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2020967730 Thanks @liurenjie1024. I read through the summary above. I think currently #245 has done the first one `Collect leave column ids after schema pruning, and translate it

Re: [PR] feat: Make OAuth token server configurable [iceberg-rust]

2024-03-26 Thread via GitHub
himadripal commented on code in PR #305: URL: https://github.com/apache/iceberg-rust/pull/305#discussion_r1539661982 ## crates/catalog/rest/src/catalog.rs: ## @@ -956,6 +964,39 @@ mod tests { ); } +#[tokio::test] +async fn test_oauth_with_auth_url() { +

Re: [I] Bug: Flink data loss after failed to refresh table [iceberg]

2024-03-26 Thread via GitHub
pvary commented on issue #9753: URL: https://github.com/apache/iceberg/issues/9753#issuecomment-2020890924 @maekchi, @Aireed: Which catalog are you using? The `SnapshotProducer` constructor uses `ops.current()` to refresh the base snapshot, like: https://github.com/apache/iceberg/

Re: [I] Add metadata tables [iceberg-python]

2024-03-26 Thread via GitHub
Fokko commented on issue #511: URL: https://github.com/apache/iceberg-python/issues/511#issuecomment-2020836310 @ndrluis certainly! 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Create table from plain Parquet files [iceberg-python]

2024-03-26 Thread via GitHub
Fokko commented on issue #445: URL: https://github.com/apache/iceberg-python/issues/445#issuecomment-2020833706 Has been fixed in https://github.com/apache/iceberg-python/pull/506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [I] Create table from plain Parquet files [iceberg-python]

2024-03-26 Thread via GitHub
Fokko closed issue #445: Create table from plain Parquet files URL: https://github.com/apache/iceberg-python/issues/445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-26 Thread via GitHub
nastra commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2020826699 > I'm just disabling link check for existing medium blog posts, which have been causing failure due to 429. WDYT? Yeah I think this is probably the best for now. Can you remove the

Re: [PR] Build: Bump Spark 3.5 to 3.5.1 [iceberg]

2024-03-26 Thread via GitHub
nastra merged PR #9832: URL: https://github.com/apache/iceberg/pull/9832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Cannot create table if location/endpoint is s3 on a "secure" Minio server [iceberg-python]

2024-03-26 Thread via GitHub
kevinjqliu commented on issue #540: URL: https://github.com/apache/iceberg-python/issues/540#issuecomment-2020817285 We can close this out. The issue is between pyiceberg and minio integration over HTTPS. We don't have a place to document minio specific behavior. -- This is an automated

Re: [I] Support Defining PartitionSpec and SortOrder without field-ids in create_table [iceberg-python]

2024-03-26 Thread via GitHub
syun64 commented on issue #338: URL: https://github.com/apache/iceberg-python/issues/338#issuecomment-2020805099 Related: https://github.com/apache/iceberg-python/pull/498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-26 Thread via GitHub
manuzhang commented on code in PR #10042: URL: https://github.com/apache/iceberg/pull/10042#discussion_r1539508033 ## site/link-checker-config.json: ## @@ -19,5 +19,8 @@ "pattern": "^../../", "replacement": "{{BASEURL}}/site/docs/" } - ] -} \ No newline at en

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-26 Thread via GitHub
manuzhang commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2020762954 I'm just disabling link check for existing medium blog posts, which have been causing failure due to 429. -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

2024-03-26 Thread via GitHub
syun64 commented on issue #368: URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2020762590 Hi @brianfromoregon and @corleyma , from my understanding of PyIceberg and PySpark Iceberg, I'm not sure if allowing the two separate clients to participate in the same transacti

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #10042: URL: https://github.com/apache/iceberg/pull/10042#discussion_r1539491468 ## site/link-checker-config.json: ## @@ -19,5 +19,8 @@ "pattern": "^../../", "replacement": "{{BASEURL}}/site/docs/" } - ] -} \ No newline at end o

Re: [I] Support reading and writing snapshot properties [iceberg-python]

2024-03-26 Thread via GitHub
syun64 closed issue #367: Support reading and writing snapshot properties URL: https://github.com/apache/iceberg-python/issues/367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Support reading and writing snapshot properties [iceberg-python]

2024-03-26 Thread via GitHub
syun64 commented on issue #367: URL: https://github.com/apache/iceberg-python/issues/367#issuecomment-2020732165 Completed by #419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Bloom filter not properly leveraged when using an OR condition [iceberg]

2024-03-26 Thread via GitHub
zhongyujiang commented on issue #10029: URL: https://github.com/apache/iceberg/issues/10029#issuecomment-2020699122 >~~If we use ParquetCombinedRowGroupFilter, for certain expressions, even if the metric filter evaluates to false, the dict filter will still be invoked, resulting in addition

Re: [PR] Build: Bump Spark 3.5 to 3.5.1 [iceberg]

2024-03-26 Thread via GitHub
nastra commented on PR #9832: URL: https://github.com/apache/iceberg/pull/9832#issuecomment-2020687347 > @nastra Is this good to go? yep, thanks for the ping. Will merge once CI passes -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Build: Bump Spark 3.5 to 3.5.1 [iceberg]

2024-03-26 Thread via GitHub
manuzhang commented on PR #9832: URL: https://github.com/apache/iceberg/pull/9832#issuecomment-2020680113 @nastra Is this good to go? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Hive: Tests to do validation hive content and iceberg table with the same name [iceberg]

2024-03-26 Thread via GitHub
nastra merged PR #9980: URL: https://github.com/apache/iceberg/pull/9980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Hive: Tests to do validation hive content and iceberg table with the same name [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #9980: URL: https://github.com/apache/iceberg/pull/9980#discussion_r1539385906 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveTableTest.java: ## @@ -349,8 +353,43 @@ public void testListTables() throws TException, IOException { H

Re: [PR] Hive: Tests to do validation hive content and iceberg table with the same name [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #9980: URL: https://github.com/apache/iceberg/pull/9980#discussion_r1539267925 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveTableTest.java: ## @@ -349,8 +353,43 @@ public void testListTables() throws TException, IOException { H

Re: [PR] Build: retry on 429 status code in link checker [iceberg]

2024-03-26 Thread via GitHub
CsengerG commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2020584976 > So maybe we should just do "aliveStatusCodes": [429, 200]? Is the order of checking the links deterministic? If it is, then that can hide issues (page is not actually present,

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-26 Thread via GitHub
viirya commented on PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#issuecomment-2020510184 @liurenjie1024 Thanks. Let me read through your summary first and explain what I've done in this PR in #244. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Manifest list encryption [iceberg]

2024-03-26 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1538622050 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +192,39 @@ private void cacheManifests(FileIO fileIO) { if (allManifests == null) {

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539276798 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -59,7 +59,7 @@ public abstract class ViewCatalogTests

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539275723 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -85,6 +85,22 @@ default Integer currentSchemaId() { @Nullable String metadataFileLocation();

Re: [PR] Hive: Tests to do validation hive content and iceberg table with the same name [iceberg]

2024-03-26 Thread via GitHub
nastra commented on code in PR #9980: URL: https://github.com/apache/iceberg/pull/9980#discussion_r1539267925 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveTableTest.java: ## @@ -349,8 +353,43 @@ public void testListTables() throws TException, IOException { H

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1538653347 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -309,65 +304,20 @@ protected enum CommitStatus { * @return Commit Status of Success

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539222675 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -191,6 +191,10 @@ protected void doCommit(TableMetadata base, TableMetadata meta

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539216264 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -113,6 +124,16 @@ public void initialize(String inputName, Map properties) { this.cl

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-26 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1539213080 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -59,7 +59,7 @@ public abstract class ViewCatalogTestshttps://github.com/apache/iceberg/pull/98

  1   2   >