Re: [PR] Run Flink, Spark3 tests on Java 17 too [iceberg]

2024-06-12 Thread via GitHub
pvary commented on PR #10477: URL: https://github.com/apache/iceberg/pull/10477#issuecomment-2162260372 > > [flink-scala-2-12-tests (8, 1.17)](https://github.com/apache/iceberg/actions/runs/9464829758/job/26072977977?pr=10477#logs) failed, not sure why. > > @pvary is your fix applica

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1635930764 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software

Re: [I] Run RevAPI without Gradle [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on issue #10368: URL: https://github.com/apache/iceberg/issues/10368#issuecomment-2162292599 I started this discussion also: https://github.com/revapi/revapi/issues/296 -- This is an automated message from the Apache Git Service. To respond to the message, please log

[I] Improve Iceberg REST S3SignRequest to include affected object [iceberg]

2024-06-12 Thread via GitHub
c-thiel opened a new issue, #10486: URL: https://github.com/apache/iceberg/issues/10486 ### Feature Request / Improvement Currently the `S3SignRequest` for remote signing for the Iceberg REST Catalog does not include the object the client wants to access explicitly (table or view uui

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-12 Thread via GitHub
HonahX commented on code in PR #807: URL: https://github.com/apache/iceberg-python/pull/807#discussion_r1635920344 ## pyiceberg/io/pyarrow.py: ## @@ -680,6 +680,10 @@ def _pyarrow_to_schema_without_ids(schema: pa.Schema) -> Schema: return visit_pyarrow(schema, _ConvertToIc

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-12 Thread via GitHub
HonahX commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2162422611 > Is this a duck db problem? Or do I need to open a new PR (from main branch + my changes) to resolve it? I tried this PR locally and did not observe this issue. My testing platform

Re: [PR] Bump msal from 1.26.0 to 1.28.0 [iceberg-python]

2024-06-12 Thread via GitHub
HonahX merged PR #812: URL: https://github.com/apache/iceberg-python/pull/812 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636077522 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636079471 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636082614 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

[I] ā“HELLLLLLLLLLLLLLLP Can not add column with the same name as a deleted and previously partitioned key column [iceberg]

2024-06-12 Thread via GitHub
madeirak opened a new issue, #10487: URL: https://github.com/apache/iceberg/issues/10487 ### Apache Iceberg version 1.4.3 ### Query engine Spark ### Please describe the bug šŸž 1. column d is also a partition field ![image](https://github.com/apache/iceber

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636134996 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Hive: Return new scan after applying column project parameter [iceberg]

2024-06-12 Thread via GitHub
pvary commented on code in PR #10449: URL: https://github.com/apache/iceberg/pull/10449#discussion_r1636140730 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -125,11 +125,9 @@ public List getSplits(JobContext context) { } String sche

Re: [PR] Docs: Add flinkVersion and flinkVersionMajor instead of hardcode [iceberg]

2024-06-12 Thread via GitHub
findepi commented on code in PR #10463: URL: https://github.com/apache/iceberg/pull/10463#discussion_r1636281141 ## docs/docs/flink-ddl.md: ## @@ -146,7 +146,7 @@ CREATE TABLE `hive_catalog`.`default`.`sample` ( ) WITH ('format-version'='2'); ``` -Table create commands suppo

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #807: URL: https://github.com/apache/iceberg-python/pull/807#discussion_r1636324658 ## pyiceberg/io/pyarrow.py: ## @@ -680,6 +680,10 @@ def _pyarrow_to_schema_without_ids(schema: pa.Schema) -> Schema: return visit_pyarrow(schema, _ConvertToIc

Re: [I] How iceberg implements materialized views [iceberg]

2024-06-12 Thread via GitHub
findepi closed issue #10406: How iceberg implements materialized views URL: https://github.com/apache/iceberg/issues/10406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] How iceberg implements materialized views [iceberg]

2024-06-12 Thread via GitHub
findepi commented on issue #10406: URL: https://github.com/apache/iceberg/issues/10406#issuecomment-2162834609 Closing as a duplicate of https://github.com/apache/iceberg/issues/10043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] MetricsReporter support close [iceberg]

2024-06-12 Thread via GitHub
huyuanfeng2018 commented on issue #9349: URL: https://github.com/apache/iceberg/issues/9349#issuecomment-2162903308 > @huyuanfeng2018 > > I see that the "closeable" trait was added in this commit, however in case of below implementations, the close() method of the MetricsReporter neve

Re: [PR] Bump azure-identity from 1.15.0 to 1.16.1 [iceberg-python]

2024-06-12 Thread via GitHub
Fokko merged PR #811: URL: https://github.com/apache/iceberg-python/pull/811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump io.delta:delta-standalone_2.12 from 3.1.0 to 3.2.0 [iceberg]

2024-06-12 Thread via GitHub
Fokko merged PR #10321: URL: https://github.com/apache/iceberg/pull/10321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Manifest list encryption [iceberg]

2024-06-12 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1636470435 ## api/src/main/java/org/apache/iceberg/Snapshot.java: ## @@ -162,6 +162,16 @@ default Iterable removedDeleteFiles(FileIO io) { */ String manifestListLocati

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1636475405 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CatalogUtils.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1636476953 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java: ## @@ -0,0 +1,294 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1636493641 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1636508431 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/SinkWriter.java: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-06-12 Thread via GitHub
ajantha-bhat commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1636542443 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/SinkWriter.java: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on PR #807: URL: https://github.com/apache/iceberg-python/pull/807#issuecomment-2163182848 > To summarize the discussion in https://github.com/apache/iceberg-python/issues/791, we could always benefit from reading data as `large_*` type since offset is 64-bit. Yes,

Re: [I] Add anchors to sections in "Configuration" documentation page [iceberg-python]

2024-06-12 Thread via GitHub
kevinjqliu commented on issue #808: URL: https://github.com/apache/iceberg-python/issues/808#issuecomment-2163223031 Ideally, all the headers and subheaders can be anchored. GitHub markdown provides this functionality: https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/

Re: [PR] Manifest list encryption [iceberg]

2024-06-12 Thread via GitHub
anuragmantri commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1636692608 ## api/src/main/java/org/apache/iceberg/Snapshot.java: ## @@ -162,6 +162,16 @@ default Iterable removedDeleteFiles(FileIO io) { */ String manifestListLocat

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
s-akhtar-baig commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1636758827 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refer

Re: [PR] Docs: Add flinkVersion and flinkVersionMajor instead of hardcode [iceberg]

2024-06-12 Thread via GitHub
manuzhang commented on code in PR #10463: URL: https://github.com/apache/iceberg/pull/10463#discussion_r1636767183 ## docs/docs/flink-ddl.md: ## @@ -146,7 +146,7 @@ CREATE TABLE `hive_catalog`.`default`.`sample` ( ) WITH ('format-version'='2'); ``` -Table create commands sup

Re: [PR] Add Files metadata table [iceberg-python]

2024-06-12 Thread via GitHub
Gowthami03B commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1636801371 ## pyiceberg/table/__init__.py: ## @@ -3537,6 +3537,106 @@ def update_partitions_map( schema=table_schema, ) +def files(self, snaps

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-12 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2163505455 I created a new PR against my fork, and once the GitHub actions failed, I manually re-tried them. https://github.com/chinmay-bhat/iceberg-python/pull/1/checks?sha=8c2e79a9c62

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1636888413 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -132,6 +138,35 @@ private Schema calculateSchema() { Types.StringTyp

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1636890810 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -132,6 +138,35 @@ private Schema calculateSchema() { Types.StringTyp

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
marvinlanhenke commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1636931579 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refe

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
marvinlanhenke commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1636931579 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refe

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
marvinlanhenke commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1636945871 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refe

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
marvinlanhenke commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1636945871 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refe

Re: [I] MetricsReporter support close [iceberg]

2024-06-12 Thread via GitHub
allen-abhishekkr commented on issue #9349: URL: https://github.com/apache/iceberg/issues/9349#issuecomment-2163881242 @huyuanfeng2018 Is there a way this can be closed? I see in other catalog files you have made changes so that close method is called. -- This is an automated message from

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
s-akhtar-baig commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1637115175 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,255 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refer

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637109731 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software F

Re: [PR] AWS: add retry logic to S3InputStream [iceberg]

2024-06-12 Thread via GitHub
puchengy commented on PR #4912: URL: https://github.com/apache/iceberg/pull/4912#issuecomment-2163990331 @amogh-jahagirdar Hi, I am checking in to see if there is any update? thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on PR #603: URL: https://github.com/apache/iceberg-python/pull/603#issuecomment-2164014600 > Thanks @syun64 for implementing this! It would be pretty useful for some work I'm doing. When do you think this will be released? I see it's on the [near term roadmap](https://githu

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-12 Thread via GitHub
stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637170443 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on PR #10020: URL: https://github.com/apache/iceberg/pull/10020#issuecomment-2164025346 Thanks, added test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1637207700 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -132,6 +138,35 @@ private Schema calculateSchema() { Types.StringType.get

Re: [I] Merge into / Upsert [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on issue #402: URL: https://github.com/apache/iceberg-python/issues/402#issuecomment-2164079178 Hi @corleyma - I opened up this PR to address your comment here by [introducing a scan API that will return a RecordBatchReader](https://github.com/apache/iceberg-python/pull/786

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
szehon-ho merged PR #10020: URL: https://github.com/apache/iceberg/pull/10020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Calling `rewrite_position_delete_files` fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
szehon-ho closed issue #9923: Calling `rewrite_position_delete_files` fails on tables with more than 1k columns URL: https://github.com/apache/iceberg/issues/9923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on PR #10020: URL: https://github.com/apache/iceberg/pull/10020#issuecomment-2164091222 Thanks @RussellSpitzer for helping get this through the finish line! PR to fix _partition metadata column collisions to come subsequently -- This is an automated message fro

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637279841 ## pyiceberg/table/__init__.py: ## @@ -454,6 +482,74 @@ def overwrite( for data_file in data_files: update_snapshot.append_dat

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637279841 ## pyiceberg/table/__init__.py: ## @@ -454,6 +482,74 @@ def overwrite( for data_file in data_files: update_snapshot.append_dat

Re: [I] Spark Dynamic Partition Pruning [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2527: URL: https://github.com/apache/iceberg/issues/2527#issuecomment-2164119585 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark Dynamic Partition Pruning [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2527: Spark Dynamic Partition Pruning URL: https://github.com/apache/iceberg/issues/2527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Flink : Data skew when we use RewriteDataFilesAction of flink to do rewrite [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2536: Flink : Data skew when we use RewriteDataFilesAction of flink to do rewrite URL: https://github.com/apache/iceberg/issues/2536 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Flink : add filters to project for flink IcebergTableSource [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2537: Flink : add filters to project for flink IcebergTableSource URL: https://github.com/apache/iceberg/issues/2537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Reduce errorprone warnings for Iceberg build [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2545: URL: https://github.com/apache/iceberg/issues/2545#issuecomment-2164119684 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: add vectorized read for flink [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2534: URL: https://github.com/apache/iceberg/issues/2534#issuecomment-2164119607 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Reduce errorprone warnings for Iceberg build [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2545: Reduce errorprone warnings for Iceberg build URL: https://github.com/apache/iceberg/issues/2545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Spark filters do not work on int96 timestamp columns [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2553: Spark filters do not work on int96 timestamp columns URL: https://github.com/apache/iceberg/issues/2553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Hive: cannot find field xxx at Map operator initialization failed when iceberg table join hive table [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2567: URL: https://github.com/apache/iceberg/issues/2567#issuecomment-2164119792 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Error when dropping partition fields for the same source field more than once. [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2569: URL: https://github.com/apache/iceberg/issues/2569#issuecomment-2164119813 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink batch job cannot read the iceberg table that writing few chang-log events in format v2. [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2571: Flink batch job cannot read the iceberg table that writing few chang-log events in format v2. URL: https://github.com/apache/iceberg/issues/2571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Support creating flink table without creating an explicit hive/hadoop catalog [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2572: URL: https://github.com/apache/iceberg/issues/2572#issuecomment-2164119874 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Hive: insert overwrite table xxx partition (pt='xxx') on iceberg table [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2576: URL: https://github.com/apache/iceberg/issues/2576#issuecomment-2164119896 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Hive: insert overwrite table xxx partition (pt='xxx') on iceberg table [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2576: Hive: insert overwrite table xxx partition (pt='xxx') on iceberg table URL: https://github.com/apache/iceberg/issues/2576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Flink: add vectorized read for flink [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2534: Flink: add vectorized read for flink URL: https://github.com/apache/iceberg/issues/2534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Flink : add filters to project for flink IcebergTableSource [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2537: URL: https://github.com/apache/iceberg/issues/2537#issuecomment-2164119648 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink : Data skew when we use RewriteDataFilesAction of flink to do rewrite [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2536: URL: https://github.com/apache/iceberg/issues/2536#issuecomment-2164119625 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Improve backward compatibility tests for spec changes introduced in all table versions [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2542: Improve backward compatibility tests for spec changes introduced in all table versions URL: https://github.com/apache/iceberg/issues/2542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Improve backward compatibility tests for spec changes introduced in all table versions [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2542: URL: https://github.com/apache/iceberg/issues/2542#issuecomment-2164119658 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark filters do not work on int96 timestamp columns [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2553: URL: https://github.com/apache/iceberg/issues/2553#issuecomment-2164119702 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Cannot Drop Table Created with HiveIcebergStorageHandler Enabled but Metadata.json is Missing [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2554: Cannot Drop Table Created with HiveIcebergStorageHandler Enabled but Metadata.json is Missing URL: https://github.com/apache/iceberg/issues/2554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Cannot Drop Table Created with HiveIcebergStorageHandler Enabled but Metadata.json is Missing [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2554: URL: https://github.com/apache/iceberg/issues/2554#issuecomment-2164119732 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Web 404 Not Found, Many page jump links cannot be opened [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2561: Web 404 Not Found, Many page jump links cannot be opened URL: https://github.com/apache/iceberg/issues/2561 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Web 404 Not Found, Many page jump links cannot be opened [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2561: URL: https://github.com/apache/iceberg/issues/2561#issuecomment-2164119746 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark on iceberg table is slower than spark on hive parquet table [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2562: URL: https://github.com/apache/iceberg/issues/2562#issuecomment-2164119755 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Hive: cannot find field xxx at Map operator initialization failed when iceberg table join hive table [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2567: Hive: cannot find field xxx at Map operator initialization failed when iceberg table join hive table URL: https://github.com/apache/iceberg/issues/2567 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Flink batch job cannot read the iceberg table that writing few chang-log events in format v2. [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] commented on issue #2571: URL: https://github.com/apache/iceberg/issues/2571#issuecomment-2164119844 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Error when dropping partition fields for the same source field more than once. [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2569: Error when dropping partition fields for the same source field more than once. URL: https://github.com/apache/iceberg/issues/2569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Support creating flink table without creating an explicit hive/hadoop catalog [iceberg]

2024-06-12 Thread via GitHub
github-actions[bot] closed issue #2572: Support creating flink table without creating an explicit hive/hadoop catalog URL: https://github.com/apache/iceberg/issues/2572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1637308620 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1637308620 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637279841 ## pyiceberg/table/__init__.py: ## @@ -454,6 +482,74 @@ def overwrite( for data_file in data_files: update_snapshot.append_dat

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-12 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1637317765 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] Can't add column with the same name as a deleted and previously partitioned key column [iceberg]

2024-06-12 Thread via GitHub
amogh-jahagirdar commented on issue #10487: URL: https://github.com/apache/iceberg/issues/10487#issuecomment-2164155791 Thanks for reporting @madeirak . I think this is similar to https://github.com/apache/iceberg/issues/10234 but this particular issue also brings up a nuance in the "unrefe

Re: [PR] Adding `add_files_overwrite` method [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #810: URL: https://github.com/apache/iceberg-python/pull/810#discussion_r1637326766 ## pyiceberg/table/__init__.py: ## @@ -474,6 +474,26 @@ def add_files(self, file_paths: List[str], snapshot_properties: Dict[str, str] = for data_file

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637279841 ## pyiceberg/table/__init__.py: ## @@ -454,6 +482,74 @@ def overwrite( for data_file in data_files: update_snapshot.append_dat

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637347402 ## pyiceberg/table/__init__.py: ## @@ -2931,14 +3161,52 @@ def _deleted_entries(self) -> List[ManifestEntry]: return [] -class OverwriteFiles(_MergingS

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-12 Thread via GitHub
liurenjie1024 commented on PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#issuecomment-2164182531 > Hi @liurenjie1024 - sorry to pester you but are you able to re-review this please? It's the last major piece of the puzzle on the read side. Hi, @sdd Sorry for delay. I'l

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637353561 ## pyiceberg/table/__init__.py: ## @@ -2931,14 +3161,52 @@ def _deleted_entries(self) -> List[ManifestEntry]: return [] -class OverwriteFiles(_MergingS

Re: [I] Can't add column with the same name as a deleted and previously partitioned key column [iceberg]

2024-06-12 Thread via GitHub
amogh-jahagirdar commented on issue #10487: URL: https://github.com/apache/iceberg/issues/10487#issuecomment-2164199701 Well probably we want to keep the historical spec around, but at least in validation we may want to skip that if itā€™s unreferenced -- This is an automated message from

Re: [PR] Support partial deletes [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1637353561 ## pyiceberg/table/__init__.py: ## @@ -2931,14 +3161,52 @@ def _deleted_entries(self) -> List[ManifestEntry]: return [] -class OverwriteFiles(_MergingS

Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]

2024-06-12 Thread via GitHub
corleyma commented on code in PR #786: URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637386352 ## pyiceberg/io/pyarrow.py: ## @@ -1005,36 +1004,46 @@ def _task_to_table( columns=[col.name for col in file_project_schema.columns], ) -

Re: [I] MetricsReporter support close [iceberg]

2024-06-12 Thread via GitHub
huyuanfeng2018 commented on issue #9349: URL: https://github.com/apache/iceberg/issues/9349#issuecomment-2164217430 > @huyuanfeng2018 Is there a way this can be closed? I see in other catalog files you have made changes so that close method is called. I think this depends on spark. Sp

Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]

2024-06-12 Thread via GitHub
corleyma commented on code in PR #786: URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637416828 ## pyiceberg/io/pyarrow.py: ## @@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT def write_parquet(

Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #786: URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637423967 ## pyiceberg/io/pyarrow.py: ## @@ -1005,36 +1004,46 @@ def _task_to_table( columns=[col.name for col in file_project_schema.columns], ) -

Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]

2024-06-12 Thread via GitHub
corleyma commented on code in PR #786: URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425676 ## pyiceberg/io/pyarrow.py: ## @@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT def write_parquet(

Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]

2024-06-12 Thread via GitHub
syun64 commented on code in PR #786: URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425908 ## pyiceberg/io/pyarrow.py: ## @@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT def write_parquet(ta

  1   2   >