Re: [PR] Fix ParallelIterable memory leak because queue continues to be added even if iterator exited [iceberg]

Heltman commented on PR #9402: URL: https://github.com/apache/iceberg/pull/9402#issuecomment-1891512061 @findepi @electrum cc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] flink has implemented the delete and update syntax support in batch mode in a later version. Will the iceberg community implement this feature [iceberg]

BlackPigHe opened a new issue, #9476: URL: https://github.com/apache/iceberg/issues/9476 ### Feature Request / Improvement flink has implemented the delete and update syntax support in batch mode in a later version. Will the iceberg community implement this feature.If iceberg is inte

[I] kerberos beeline insert iceberg fail error: Job commit failed: org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore [iceberg]

xiaolan-bit opened a new issue, #9475: URL: https://github.com/apache/iceberg/issues/9475 ### Apache Iceberg version 1.3.1 ### Query engine Hive ### Please describe the bug 🐞 *version: hive-3.1.3 iceberg-1.3.1 kerberos-1.15.1 hadoop-3.3.6 user: hadoop

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

stevenzwu commented on code in PR #9452: URL: https://github.com/apache/iceberg/pull/9452#discussion_r1451920250 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/util/FlinkPackage.java: ## @@ -19,15 +19,31 @@ package org.apache.iceberg.flink.util; import org.apach

Re: [PR] Spark: Fix SparkTable to use name and effective snapshotID for comparing [iceberg]

wooyeong commented on PR #9455: URL: https://github.com/apache/iceberg/pull/9455#issuecomment-1891165135 @ajantha-bhat gentle ping, could you review this change, please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] On iceberg11 getting s3 connection reset error [iceberg]

javrasya commented on issue #4457: URL: https://github.com/apache/iceberg/issues/4457#issuecomment-1891155838 I am using Flink to consume from a table which gets upserts and I am getting this error too. Tried tweaking http client socket timeouts and stuff but nothing has worked so for for m

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451838385 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451838116 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451837941 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451837579 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451837508 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451836798 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451836277 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451836112 ## pyiceberg/table/__init__.py: ## @@ -1910,3 +2006,137 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451835863 ## pyiceberg/table/__init__.py: ## @@ -831,6 +887,46 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451834512 ## pyiceberg/io/pyarrow.py: ## @@ -1565,13 +1564,54 @@ def fill_parquet_file_metadata( del upper_bounds[field_id] del null_value_counts[field_id]

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451834512 ## pyiceberg/io/pyarrow.py: ## @@ -1565,13 +1564,54 @@ def fill_parquet_file_metadata( del upper_bounds[field_id] del null_value_counts[field_id]

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451833666 ## pyiceberg/io/pyarrow.py: ## @@ -1565,13 +1565,56 @@ def fill_parquet_file_metadata( del upper_bounds[field_id] del null_value_counts[field_id]

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451832516 ## pyiceberg/table/__init__.py: ## @@ -831,6 +887,46 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [I] [DOC] Reorder pages under Spark in the nav bar [iceberg]

github-actions[bot] commented on issue #6724: URL: https://github.com/apache/iceberg/issues/6724#issuecomment-1891124149 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] How to detect if the partition's data is ready to consume [iceberg]

github-actions[bot] commented on issue #6725: URL: https://github.com/apache/iceberg/issues/6725#issuecomment-1891124143 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Add checkstyle rule to ensure AssertJ assertions always check for underlying exception message [iceberg]

github-actions[bot] commented on issue #7040: URL: https://github.com/apache/iceberg/issues/7040#issuecomment-1891124126 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451832048 ## pyiceberg/table/__init__.py: ## @@ -831,6 +887,46 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [I] How to detect if the partition's data is ready to consume [iceberg]

github-actions[bot] closed issue #6725: How to detect if the partition's data is ready to consume URL: https://github.com/apache/iceberg/issues/6725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451831352 ## pyiceberg/table/__init__.py: ## @@ -831,6 +887,46 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451829404 ## pyiceberg/table/__init__.py: ## @@ -797,6 +850,9 @@ def location(self) -> str: def last_sequence_number(self) -> int: return self.metadata.last_seque

Re: [PR] Write support [iceberg-python]

rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1451809069 ## pyiceberg/io/pyarrow.py: ## @@ -1565,13 +1564,54 @@ def fill_parquet_file_metadata( del upper_bounds[field_id] del null_value_counts[field_id]

Re: [PR] Parquet: Add system config for unsafe Parquet ID fallback. [iceberg]

rdblue commented on PR #9324: URL: https://github.com/apache/iceberg/pull/9324#issuecomment-1891081451 @jackye1995, @danielcweeks, @RussellSpitzer, could you look at this? I'd like to ideally get it into the next release since we have been allowing unsafe reads. -- This is an automated m

Re: [PR] Avro data encryption [iceberg]

rdblue commented on PR #9436: URL: https://github.com/apache/iceberg/pull/9436#issuecomment-1891079101 Looks like this is ready to commit when it is rebased on top of the AES GCM stream changes. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] AES GCM Stream changes [iceberg]

rdblue merged PR #9453: URL: https://github.com/apache/iceberg/pull/9453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451801651 ## kafka-connect/build.gradle: ## @@ -30,3 +30,30 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect-events") { useJUnitPlatform() } } + +project(":ice

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451801429 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -0,0 +1,478 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451800872 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordWrapper.java: ## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451800701 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordWriter.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451800555 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,511 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451800122 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/PartitionedAppendWriter.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451799486 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451799323 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451799004 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451798927 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451798132 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451797484 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451797289 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451797043 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451796523 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

rdblue commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1451796402 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] support python 3.12 [iceberg-python]

MehulBatra commented on code in PR #254: URL: https://github.com/apache/iceberg-python/pull/254#discussion_r1451758589 ## pyproject.toml: ## @@ -70,6 +71,9 @@ adlfs = { version = ">=2023.1.0,<2024.1.0", optional = true } gcsfs = { version = ">=2023.1.0,<2024.1.0", optional = tr

Re: [PR] support python 3.12 [iceberg-python]

MehulBatra commented on code in PR #254: URL: https://github.com/apache/iceberg-python/pull/254#discussion_r1451758129 ## pyproject.toml: ## @@ -29,7 +29,8 @@ classifiers = [ "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming

Re: [PR] Add SqlCatalog _commit_table support [iceberg-python]

syun64 commented on code in PR #265: URL: https://github.com/apache/iceberg-python/pull/265#discussion_r1451754672 ## pyiceberg/catalog/sql.py: ## @@ -268,16 +269,32 @@ def drop_table(self, identifier: Union[str, Identifier]) -> None: identifier_tuple = self.identifier

Re: [PR] Add SqlCatalog _commit_table support [iceberg-python]

HonahX commented on code in PR #265: URL: https://github.com/apache/iceberg-python/pull/265#discussion_r1451680767 ## pyiceberg/catalog/sql.py: ## @@ -268,16 +269,32 @@ def drop_table(self, identifier: Union[str, Identifier]) -> None: identifier_tuple = self.identifier

Re: [PR] Add SqlCatalog _commit_table support [iceberg-python]

HonahX commented on code in PR #265: URL: https://github.com/apache/iceberg-python/pull/265#discussion_r1451680767 ## pyiceberg/catalog/sql.py: ## @@ -268,16 +269,32 @@ def drop_table(self, identifier: Union[str, Identifier]) -> None: identifier_tuple = self.identifier

Re: [PR] Add SqlCatalog _commit_table support [iceberg-python]

HonahX commented on code in PR #265: URL: https://github.com/apache/iceberg-python/pull/265#discussion_r1451680767 ## pyiceberg/catalog/sql.py: ## @@ -268,16 +269,32 @@ def drop_table(self, identifier: Union[str, Identifier]) -> None: identifier_tuple = self.identifier

Re: [PR] Add SqlCatalog _commit_table support [iceberg-python]

HonahX commented on code in PR #265: URL: https://github.com/apache/iceberg-python/pull/265#discussion_r1451680767 ## pyiceberg/catalog/sql.py: ## @@ -268,16 +269,32 @@ def drop_table(self, identifier: Union[str, Identifier]) -> None: identifier_tuple = self.identifier