Re: [PR] Flink - Fix incorrect / old row being written into delta files when using upsert mode [iceberg]

pvary commented on code in PR #4364: URL: https://github.com/apache/iceberg/pull/4364#discussion_r1744988793 ## flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/BaseDeltaTaskWriter.java: ## @@ -74,7 +80,7 @@ public void write(RowData row) throws IOException {

Re: [I] Why call deleteKey for Insert and Update After in Flink BaseDeltaTaskWriter? [iceberg]

SML0127 commented on issue #11081: URL: https://github.com/apache/iceberg/issues/11081#issuecomment-2330951529 @pvary Thk for answer pvary. Now I'm checking some code in ColumnarBatchReader, whether it works as follow: 1. apply eq delete files first, from oldest snapshot 2

Re: [I] Flink: multiple sinks for the different iceberg tables in the same job？ [iceberg]

chenwyi2 closed issue #11074: Flink: multiple sinks for the different iceberg tables in the same job？ URL: https://github.com/apache/iceberg/issues/11074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Flink: multiple sinks for the different iceberg tables in the same job？ [iceberg]

chenwyi2 commented on issue #11074: URL: https://github.com/apache/iceberg/issues/11074#issuecomment-2331046697 yes that's no iceberg problem, but with our platform, thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[I] Caused by: java.lang.RuntimeException: org.apache.flink.runtime.JobException: Creating the input splits caused an error: Failed to refresh the table [iceberg]

Littlehhao opened a new issue, #11082: URL: https://github.com/apache/iceberg/issues/11082 ### Query engine _No response_ ### Question environment: flink-standlone:1.17.1 hadoop 3.1.0 select * from sample; Caused by: org.apache.flink.runtime.client

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

ajantha-bhat commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1745249956 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includ

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

ajantha-bhat commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1745249956 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includ

Re: [I] Why call deleteKey for Insert and Update After in Flink BaseDeltaTaskWriter? [iceberg]

SML0127 commented on issue #11081: URL: https://github.com/apache/iceberg/issues/11081#issuecomment-2331421137 @pvary Sorry for asking a vague question. But thx to your answer, now I understand how data file and delete file works. That is, - equality delete files are valid only

Re: [PR] feat: support projection pushdown for datafusion iceberg [iceberg-rust]

FANNG1 commented on code in PR #594: URL: https://github.com/apache/iceberg-rust/pull/594#discussion_r1745529020 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -138,3 +156,18 @@ async fn get_batch_stream( Ok(Box::pin(stream)) } + +fn get_column_names(

Re: [PR] feat: support projection pushdown for datafusion iceberg [iceberg-rust]

FANNG1 commented on PR #594: URL: https://github.com/apache/iceberg-rust/pull/594#issuecomment-2331731075 @liurenjie1024 , all comments are addressed, please help to review again, thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Kafka Connect: Docs on configuring the sink [iceberg]

bryanck commented on code in PR #10746: URL: https://github.com/apache/iceberg/pull/10746#discussion_r1745578149 ## docs/docs/kafka-connect.md: ## @@ -0,0 +1,352 @@ +--- +title: "Kafka Connect" +--- + + +# Kafka Connect + +[Kafka Connect](https://docs.confluent.io/platform/curre

Re: [PR] Kafka Connect: increase timeout for integration test [iceberg]

bryanck commented on PR #11075: URL: https://github.com/apache/iceberg/pull/11075#issuecomment-2331804459 > Can we create a separate CI for kafka connect? Sure let me add that to this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Bump cryptography from 43.0.0 to 43.0.1 [iceberg-python]

Fokko merged PR #1130: URL: https://github.com/apache/iceberg-python/pull/1130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Bump mkdocs-material from 9.5.33 to 9.5.34 [iceberg-python]

Fokko merged PR #1126: URL: https://github.com/apache/iceberg-python/pull/1126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Make `commit_table` public [iceberg-python]

Fokko commented on PR #1112: URL: https://github.com/apache/iceberg-python/pull/1112#issuecomment-2331938676 @sungwy No problem at all, I've pulled in latest master 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows closed pull request #10188: MR: iceberg storage handler should set common projection pruning config URL: https://github.com/apache/iceberg/pull/10188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] open-api: Fix compile warnings for testFixtures [iceberg]

danielcweeks merged PR #11071: URL: https://github.com/apache/iceberg/pull/11071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] Spark 3.5: Mandate identifier fields when create_changelog_view for table contain unsortable columns [iceberg]

dramaticlly commented on code in PR #11045: URL: https://github.com/apache/iceberg/pull/11045#discussion_r1745898693 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/CreateChangelogViewProcedure.java: ## @@ -146,10 +147,16 @@ public InternalRow[] call(Intern

Re: [PR] Spark 3.3, 3.4: Parallelize reading files in migrate procedures [iceberg]

amogh-jahagirdar commented on PR #11043: URL: https://github.com/apache/iceberg/pull/11043#issuecomment-2332252827 Thanks @manuzhang! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spark 3.3, 3.4: Parallelize reading files in migrate procedures [iceberg]

amogh-jahagirdar merged PR #11043: URL: https://github.com/apache/iceberg/pull/11043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Rust <> Python integration point [iceberg-rust]

kevinjqliu commented on issue #538: URL: https://github.com/apache/iceberg-rust/issues/538#issuecomment-2332259085 Looks like @sungwy already started by exposing Transforms in #556 I'll take a stab at exposing the Catalogs, see https://github.com/apache/iceberg-rust/pull/534#issuecom

Re: [PR] Support python 3.12 [iceberg-python]

Fokko commented on code in PR #1068: URL: https://github.com/apache/iceberg-python/pull/1068#discussion_r1745925508 ## pyproject.toml: ## @@ -611,6 +615,8 @@ filterwarnings = [ "ignore:unclosed

Re: [I] Add support for Python 3.12 [iceberg-python]

kevinjqliu closed issue #28: Add support for Python 3.12 URL: https://github.com/apache/iceberg-python/issues/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

danielcweeks commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1745961589 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includ

Re: [PR] Spark 3.5: Mandate identifier fields when create_changelog_view for table contain unsortable columns [iceberg]

flyrain merged PR #11045: URL: https://github.com/apache/iceberg/pull/11045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spark 3.5: Mandate identifier fields when create_changelog_view for table contain unsortable columns [iceberg]

flyrain commented on PR #11045: URL: https://github.com/apache/iceberg/pull/11045#issuecomment-2332359477 Thanks @dramaticlly for the change, and thanks @karuppayya @anigos @huaxingao for the review. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Kafka Connect: Include third party licenses and notices in distribution [iceberg]

bryanck commented on code in PR #10829: URL: https://github.com/apache/iceberg/pull/10829#discussion_r1745974177 ## kafka-connect/kafka-connect-runtime/NOTICE: ## @@ -0,0 +1,1723 @@ + +Apache Iceberg +Copyright 2017-2024 The Apache Software Foundation + +This product includes so

[PR] Expose `Catalog` trait as python binding [iceberg-rust]

kevinjqliu opened a new pull request, #604: URL: https://github.com/apache/iceberg-rust/pull/604 #538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] New PR label [ready for review] [iceberg-python]

sungwy commented on issue #1123: URL: https://github.com/apache/iceberg-python/issues/1123#issuecomment-2332404830 As a counter argument, what's the difference between using this label, versus opening a PR in Draft mode? -- This is an automated message from the Apache Git Service. To res

Re: [I] Cannot commit identity partition on datatypes time,timestamp* using 'fromPartitionString' [iceberg]

mderoy commented on issue #11085: URL: https://github.com/apache/iceberg/issues/11085#issuecomment-2332466946 We'll contribute on this in a few weeks when one of our developers gets back from vacation. https://github.com/apache/iceberg/commit/770f84325f2810bae3b48a6e1983d6a8135cb7bf see

[PR] Remove deprecated `datetime` functions [iceberg-python]

hussein-awala opened a new pull request, #1134: URL: https://github.com/apache/iceberg-python/pull/1134 closes: #1133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Inconsistent row count across versions [iceberg-python]

dev-goyal commented on issue #1132: URL: https://github.com/apache/iceberg-python/issues/1132#issuecomment-2332547741 Thanks @sungwy , that makes sense to me - I am indeed using MOR (version 2), so this makes sense to me! Let me know how else I might be able to help. -- This is an automa

[PR] Core: Optimize writing metadata for many new files [iceberg]

aokolnychyi opened a new pull request, #11086: URL: https://github.com/apache/iceberg/pull/11086 This PR optimizes writing metadata for many new files, which is helpful during initial table creation and row-level operations that modify many files. Benchmark results prior to the change

Re: [PR] Core: Optimize writing metadata for many new files [iceberg]

aokolnychyi commented on code in PR #11086: URL: https://github.com/apache/iceberg/pull/11086#discussion_r1746178142 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -654,4 +740,38 @@ private static void updateTotal( } } } + + protected stati

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

amogh-jahagirdar commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1746200353 ## core/src/test/java/org/apache/iceberg/TestRewriteFiles.java: ## @@ -384,6 +386,116 @@ public void testRewriteDataAndAssignOldSequenceNumber() { asser

Re: [I] Cannot commit identity partition on datatypes time,timestamp* using 'fromPartitionString' [iceberg]

amogh-jahagirdar commented on issue #11085: URL: https://github.com/apache/iceberg/issues/11085#issuecomment-2332705117 Hey @mderoy check out https://github.com/apache/iceberg/pull/10820 which tried to address this but we determined that adding to this Conversions logic is probably not the

Re: [I] New PR label [ready for review] [iceberg-python]

kevinjqliu closed issue #1123: New PR label [ready for review] URL: https://github.com/apache/iceberg-python/issues/1123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Remove python 3.8 support [iceberg-python]

kevinjqliu commented on issue #1121: URL: https://github.com/apache/iceberg-python/issues/1121#issuecomment-2332722899 I'll draft it up :) thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Deprecate ADLFS prefix in favor of ADLS [iceberg-python]

kevinjqliu commented on PR #961: URL: https://github.com/apache/iceberg-python/pull/961#issuecomment-2332733081 @ndrluis do you mind rebasing this PR? Looks like its almost good to go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[PR] Bump pydantic from 2.8.2 to 2.9.0 [iceberg-python]

dependabot[bot] opened a new pull request, #1137: URL: https://github.com/apache/iceberg-python/pull/1137 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.8.2 to 2.9.0. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

[PR] Bump sqlalchemy from 2.0.32 to 2.0.34 [iceberg-python]

dependabot[bot] opened a new pull request, #1138: URL: https://github.com/apache/iceberg-python/pull/1138 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.32 to 2.0.34. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's

Re: [I] Support to optimize, analyze tables and expire snapshots, remove orphan files [iceberg-python]

eedduuar commented on issue #31: URL: https://github.com/apache/iceberg-python/issues/31#issuecomment-2332788362 Hello, any progress? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Improve Documentation on getting started with GCS [iceberg]

github-actions[bot] commented on issue #7948: URL: https://github.com/apache/iceberg/issues/7948#issuecomment-2332934209 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Avoid concurrent commits causing commit failures [iceberg]

github-actions[bot] commented on PR #8001: URL: https://github.com/apache/iceberg/pull/8001#issuecomment-2332935581 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] metadata.json delete [iceberg]

github-actions[bot] commented on issue #8007: URL: https://github.com/apache/iceberg/issues/8007#issuecomment-2332935606 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Write ordered by within unique physical partitions folder (exclude hash path). [iceberg]

github-actions[bot] commented on issue #8008: URL: https://github.com/apache/iceberg/issues/8008#issuecomment-2332935627 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] remove_orphan_files throws reached maximum depth exception in AWS EMR-6.11.0 [iceberg]

github-actions[bot] commented on issue #8022: URL: https://github.com/apache/iceberg/issues/8022#issuecomment-2332935648 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] [spark 3.4] skip empty file during table migration, table snapshotting or adding files [iceberg]

github-actions[bot] commented on PR #8040: URL: https://github.com/apache/iceberg/pull/8040#issuecomment-2332935751 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Spark 3.3: Adding Rebalance operator solving for small files problem [iceberg]

github-actions[bot] commented on PR #8042: URL: https://github.com/apache/iceberg/pull/8042#issuecomment-2332935802 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] AWS: Add retry on UncheckedIOException and max retries for S3FileIO [iceberg]

github-actions[bot] commented on PR #8043: URL: https://github.com/apache/iceberg/pull/8043#issuecomment-2332935830 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

[PR] Docs: Document accessing instance variables [iceberg]

aokolnychyi opened a new pull request, #11087: URL: https://github.com/apache/iceberg/pull/11087 This PR documents our recommendations for accessing instance variables to improve consistency of the code. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Pyiceberg StaticTable use the last metadata json URL when the full path is not provided [iceberg]

djouallah closed issue #7979: Pyiceberg StaticTable use the last metadata json URL when the full path is not provided URL: https://github.com/apache/iceberg/issues/7979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark: Deprecate SparkAppenderFactory [iceberg]

ajantha-bhat commented on PR #11076: URL: https://github.com/apache/iceberg/pull/11076#issuecomment-2332990957 @amoghjaha-db: > Since the class is package private do we want to just remove it upfront I did get this thought initially and checked how we handled previously for

[PR] add more tests for position deletes [iceberg-python]

sungwy opened a new pull request, #1141: URL: https://github.com/apache/iceberg-python/pull/1141 Investigating #1132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] API, Core: Add manifestPath API to ContentFile which will return the path to a manifest from which the content file resides in [iceberg]

aokolnychyi commented on code in PR #11044: URL: https://github.com/apache/iceberg/pull/11044#discussion_r1746373228 ## core/src/main/java/org/apache/iceberg/InheritableMetadataFactory.java: ## @@ -46,11 +49,14 @@ static class BaseInheritableMetadata implements InheritableMetad

Re: [PR] API, Core: Add manifestPath API to ContentFile which will return the path to a manifest from which the content file resides in [iceberg]

aokolnychyi commented on code in PR #11044: URL: https://github.com/apache/iceberg/pull/11044#discussion_r1746373228 ## core/src/main/java/org/apache/iceberg/InheritableMetadataFactory.java: ## @@ -46,11 +49,14 @@ static class BaseInheritableMetadata implements InheritableMetad

Re: [PR] TableMetadataBuilder [iceberg-rust]

liurenjie1024 commented on PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#issuecomment-2333006736 Thanks @c-thiel for this pr, I've skimmed through it and it looks great to me. However this pr is too huge to review(3k lines), would you mind to split them into smaller onces? F

Re: [I] Discussion: Typesafe(r) properties [iceberg-rust]

liurenjie1024 commented on issue #599: URL: https://github.com/apache/iceberg-rust/issues/599#issuecomment-2333037728 Thanks @c-thiel for raising this. I love this idea of type safe properties and believe this is the right direction to go. I took a look at your reference, but I didn't get a

Re: [I] Why call deleteKey for Insert and Update After in Flink BaseDeltaTaskWriter? [iceberg]

SML0127 commented on issue #11081: URL: https://github.com/apache/iceberg/issues/11081#issuecomment-2333065332 @pvary Thank you for your support. It really helped me a lot. Lastly, I have one more question. I am looking at the code `ColumnarBatchReader.java`. Where can I see that the

Re: [PR] Docs: Document accessing instance variables [iceberg]

manuzhang commented on code in PR #11087: URL: https://github.com/apache/iceberg/pull/11087#discussion_r1746409783 ## site/docs/contribute.md: ## @@ -388,6 +388,34 @@ When passing boolean arguments to existing or external methods, use inline comme dropTable(identifier, purge

Re: [PR] feat: implement IcebergTableProviderFactory for datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #600: URL: https://github.com/apache/iceberg-rust/pull/600#discussion_r1746464932 ## crates/integrations/datafusion/src/table/table_provider_factory.rs: ## @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +//

Re: [PR] feat: implement IcebergTableProviderFactory for datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #600: URL: https://github.com/apache/iceberg-rust/pull/600#discussion_r1746466160 ## crates/integrations/datafusion/src/table/table_provider_factory.rs: ## @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +//

Re: [PR] Docs: Document accessing instance variables [iceberg]

aokolnychyi commented on code in PR #11087: URL: https://github.com/apache/iceberg/pull/11087#discussion_r1746503833 ## site/docs/contribute.md: ## @@ -388,6 +388,34 @@ When passing boolean arguments to existing or external methods, use inline comme dropTable(identifier, pur

Re: [PR] Table Scan: Add Row Selection Filtering [iceberg-rust]

sdd commented on code in PR #565: URL: https://github.com/apache/iceberg-rust/pull/565#discussion_r1746573022 ## crates/iceberg/src/expr/visitors/page_index_evaluator.rs: ## @@ -0,0 +1,1491 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Table Scan: Add Row Selection Filtering [iceberg-rust]

sdd commented on code in PR #565: URL: https://github.com/apache/iceberg-rust/pull/565#discussion_r1746597564 ## crates/iceberg/src/expr/visitors/page_index_evaluator.rs: ## @@ -0,0 +1,1491 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor