Re: [I] Kafka Connect: Record projection Index out of bounds error [iceberg]

2024-09-08 Thread via GitHub
ismailsimsek commented on issue #11099: URL: https://github.com/apache/iceberg/issues/11099#issuecomment-2336596778 Just found the issue! it was the issue when generating `GenericAppenderFactory` in which full table schema was given, instead of key schema https://github.com/tabular-

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.30 to 1.27.33 [iceberg-go]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #140: URL: https://github.com/apache/iceberg-go/pull/140 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.30 to 1.27.33. Commits https://github.com/aws/aws-sdk-go-v2/commit/f1d71c59a149998187

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 [iceberg-go]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #141: URL: https://github.com/apache/iceberg-go/pull/141 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.30.4 to 1.30.5. Commits https://github.com/aws/aws-sdk-go-v2/commit/a2b751d1ba71f59175a41f9cae5f

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.32 [iceberg-go]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #143: URL: https://github.com/apache/iceberg-go/pull/143 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.29 to 1.17.32. Commits https://github.com/aws/aws-sdk-go-v2/commit/f1d71c59a1499

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.30 [iceberg-go]

2024-09-08 Thread via GitHub
dependabot[bot] closed pull request #138: build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.30 URL: https://github.com/apache/iceberg-go/pull/138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.30 [iceberg-go]

2024-09-08 Thread via GitHub
dependabot[bot] commented on PR #138: URL: https://github.com/apache/iceberg-go/pull/138#issuecomment-2336599806 Superseded by #143. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] feat: SortOrder methods should take schema ref if possible [iceberg-rust]

2024-09-08 Thread via GitHub
c-thiel opened a new pull request, #613: URL: https://github.com/apache/iceberg-rust/pull/613 Reduce clones I also added two tests that didn't fail but where missing IMO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[PR] Feat: Reassign field ids for schema [iceberg-rust]

2024-09-08 Thread via GitHub
c-thiel opened a new pull request, #615: URL: https://github.com/apache/iceberg-rust/pull/615 Required for `TableMetadataBuilder` (https://github.com/apache/iceberg-rust/pull/587). When new TableMetadata is created, field ids should be reassign to ensure that field numbering is norma

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-09-08 Thread via GitHub
c-thiel commented on PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#issuecomment-2336618257 @liurenjie1024 I tried to cut a few things out - but not along the lines of `TalbeUpdate`. I hope that's OK? 1. https://github.com/apache/iceberg-rust/pull/611 2. https://gith

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749162764 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749162764 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749163419 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749163419 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749163419 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749163419 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#issuecomment-2336631735 Thank you very much for the review and comments, @liurenjie1024 I think I have addressed all your comments and questions and would be happy if you can take a another look. I

[PR] chore(deps): Bump crate-ci/typos from 1.24.3 to 1.24.5 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #616: URL: https://github.com/apache/iceberg-rust/pull/616 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. Release notes Sourced from https://github.com/crate-ci/typos/releases";>crate-ci/typos's releases.

[PR] chore(deps): Update arrow-array requirement from 52 to 53 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #617: URL: https://github.com/apache/iceberg-rust/pull/617 Updates the requirements on [arrow-array](https://github.com/apache/arrow-rs) to permit the latest version. Changelog Sourced from https://github.com/apache/arrow-rs/blob/master/CH

[PR] chore(deps): Update parquet requirement from 52 to 53 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] opened a new pull request, #619: URL: https://github.com/apache/iceberg-rust/pull/619 Updates the requirements on [parquet](https://github.com/apache/arrow-rs) to permit the latest version. Changelog Sourced from https://github.com/apache/arrow-rs/blob/master/CHANGE

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-08 Thread via GitHub
a-agmon commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1749162764 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-09-08 Thread via GitHub
manuzhang commented on PR #9841: URL: https://github.com/apache/iceberg/pull/9841#issuecomment-2336715251 As a newbie to the native land, I'm wondering why we integrate with Comet parquet reader here. Any alternatives? Is there an "official" parquet reader? -- This is an automated message

Re: [PR] feat: SortOrder methods should take schema ref if possible [iceberg-rust]

2024-09-08 Thread via GitHub
Xuanwo commented on code in PR #613: URL: https://github.com/apache/iceberg-rust/pull/613#discussion_r1749260477 ## crates/iceberg/src/spec/sort.rs: ## @@ -133,6 +133,14 @@ impl SortOrder { pub fn is_unsorted(&self) -> bool { self.fields.is_empty() } + +//

[PR] fix: SIGSEGV when describe empty table [iceberg-go]

2024-09-08 Thread via GitHub
alex-kar opened a new pull request, #145: URL: https://github.com/apache/iceberg-go/pull/145 Empty table does not have "Current Snapshot" causing describe command to be failed. Added validation to replace with empty string `""`. Steps to reproduce 1. Start env `docker-compos

Re: [PR] feat: partition compatibility [iceberg-rust]

2024-09-08 Thread via GitHub
c-thiel commented on code in PR #612: URL: https://github.com/apache/iceberg-rust/pull/612#discussion_r1749274990 ## crates/iceberg/src/spec/partition.rs: ## @@ -118,9 +118,63 @@ impl PartitionSpec { /// Turn this partition spec into an unbound partition spec. ///

Re: [PR] feat: partition compatibility [iceberg-rust]

2024-09-08 Thread via GitHub
Xuanwo merged PR #612: URL: https://github.com/apache/iceberg-rust/pull/612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Feat: Normalize TableMetadata [iceberg-rust]

2024-09-08 Thread via GitHub
Xuanwo commented on code in PR #611: URL: https://github.com/apache/iceberg-rust/pull/611#discussion_r1749292893 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -278,6 +330,229 @@ impl TableMetadata { self.snapshots .insert(snapshot.snapshot_id(), Arc::

Re: [I] Spark Streaming Job with multiple queries MERGE INTO the same target table (Runtime file filtering is not possible) [iceberg]

2024-09-08 Thread via GitHub
eric-maynard commented on issue #11094: URL: https://github.com/apache/iceberg/issues/11094#issuecomment-2336763248 To use multiple sessions you would essentially move this part of your code into the `start_streaming_query` function: ``` # Initialize Spark Session spark = ( Spar

[PR] add option to force columns to lowercase [iceberg]

2024-09-08 Thread via GitHub
yornstei opened a new pull request, #11100: URL: https://github.com/apache/iceberg/pull/11100 The current connector will create the columns with the same casing as the incoming data, even though many catalogs will show the columns as lowercase. This PR will give the option to force columns

Re: [PR] feat: Reassign field ids for schema [iceberg-rust]

2024-09-08 Thread via GitHub
Fokko commented on code in PR #615: URL: https://github.com/apache/iceberg-rust/pull/615#discussion_r1749313279 ## crates/iceberg/src/spec/schema.rs: ## @@ -86,6 +87,16 @@ impl SchemaBuilder { self } +/// Reassign all field-ids (nested) on build. +/// If

Re: [I] storage/s3: We should respect `client.region` too [iceberg-rust]

2024-09-08 Thread via GitHub
jdockerty commented on issue #603: URL: https://github.com/apache/iceberg-rust/issues/603#issuecomment-2336810234 I can do this đź‘Ť -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Spark 3.3,3.4: Add back log making it clearer when we are not pushing down filters [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] closed pull request #7713: Spark 3.3,3.4: Add back log making it clearer when we are not pushing down filters URL: https://github.com/apache/iceberg/pull/7713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on PR #7732: URL: https://github.com/apache/iceberg/pull/7732#issuecomment-2336880892 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark 3.4: Correct the two-stage parsing strategy of antlr parser [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on PR #7734: URL: https://github.com/apache/iceberg/pull/7734#issuecomment-2336880932 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark 3.4: Correct the two-stage parsing strategy of antlr parser [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] closed pull request #7734: Spark 3.4: Correct the two-stage parsing strategy of antlr parser URL: https://github.com/apache/iceberg/pull/7734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] closed pull request #7790: Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf URL: https://github.com/apache/iceberg/pull/7790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on PR #7790: URL: https://github.com/apache/iceberg/pull/7790#issuecomment-2336880984 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Disable containerReuse when building delete filters [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on PR #7833: URL: https://github.com/apache/iceberg/pull/7833#issuecomment-2336881035 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: support use-table-distribution-and-ordering in session conf [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on PR #8164: URL: https://github.com/apache/iceberg/pull/8164#issuecomment-2336881202 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Case sensitivity is not respected when using IcebergGenerics.ScanBuilder [iceberg]

2024-09-08 Thread via GitHub
github-actions[bot] commented on issue #8178: URL: https://github.com/apache/iceberg/issues/8178#issuecomment-2336881273 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support metadata compaction [iceberg-python]

2024-09-08 Thread via GitHub
github-actions[bot] closed issue #270: Support metadata compaction URL: https://github.com/apache/iceberg-python/issues/270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Support metadata compaction [iceberg-python]

2024-09-08 Thread via GitHub
github-actions[bot] commented on issue #270: URL: https://github.com/apache/iceberg-python/issues/270#issuecomment-2336882412 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [PR] chore(deps): Update arrow-array requirement from 52 to 53 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] commented on PR #617: URL: https://github.com/apache/iceberg-rust/pull/617#issuecomment-2336926987 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Update arrow-ord requirement from 52 to 53 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] commented on PR #621: URL: https://github.com/apache/iceberg-rust/pull/621#issuecomment-2336927231 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Update arrow-arith requirement from 52 to 53 [iceberg-rust]

2024-09-08 Thread via GitHub
dependabot[bot] commented on PR #620: URL: https://github.com/apache/iceberg-rust/pull/620#issuecomment-2336927697 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [I] storage/s3: We should respect `client.region` too [iceberg-rust]

2024-09-08 Thread via GitHub
Xuanwo closed issue #603: storage/s3: We should respect `client.region` too URL: https://github.com/apache/iceberg-rust/issues/603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] fix: Correctly calculate highest_field_id in schema [iceberg-rust]

2024-09-08 Thread via GitHub
Xuanwo merged PR #590: URL: https://github.com/apache/iceberg-rust/pull/590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-09-08 Thread via GitHub
huaxingao commented on PR #9841: URL: https://github.com/apache/iceberg/pull/9841#issuecomment-2337062384 @manuzhang We have native Parquet reader on Comet side to take advantage of the performance gain on native side. -- This is an automated message from the Apache Git Service. To respon

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-09-08 Thread via GitHub
liurenjie1024 commented on PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#issuecomment-2337063983 > After we answered those questions, and we still think splitting makes sense, I can try to find time to build stacked-PRs. Maybe just splitting normalization / validation in tab

Re: [I] Support relative paths in Table Metadata [iceberg]

2024-09-08 Thread via GitHub
lightmelodies commented on issue #1617: URL: https://github.com/apache/iceberg/issues/1617#issuecomment-2337073064 For anyone who is interested: https://github.com/lightmelodies/iceberg-relative-io -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Core: Parallelize manifest writing for many new files [iceberg]

2024-09-08 Thread via GitHub
stevenzwu commented on code in PR #11086: URL: https://github.com/apache/iceberg/pull/11086#discussion_r1749541033 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -42,6 +42,25 @@ protected static List parameters() { return Arrays.asList(1, 2, 3); }

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-09-08 Thread via GitHub
manuzhang commented on PR #9841: URL: https://github.com/apache/iceberg/pull/9841#issuecomment-2337144632 @huaxingao I know that. What about alternatives? Is the Comet parquet reader the only native parquet reader implementation? -- This is an automated message from the Apache Git Service

Re: [PR] Spec: Fix rendering of partition stats file spec [iceberg]

2024-09-08 Thread via GitHub
nastra merged PR #11068: URL: https://github.com/apache/iceberg/pull/11068 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap