Re: [PR] Concurrent table scans [iceberg-rust]

2024-06-13 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1639361290 ## crates/iceberg/src/scan.rs: ## @@ -302,13 +262,147 @@ impl TableScan { arrow_reader_builder.build().read(self.plan_files().await?) } +} + +#[derive(De

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
gyfora commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639299474 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
sdd commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2167261204 Thanks for this @ZENOTME. The first diagram in your comment above is exactly what I'm currently working to implement as part of a service that exposes an Apache Arrow Flight interface

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639292398 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639284858 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639281217 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639280951 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639276069 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639268169 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-13 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639261053 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java: ## @@ -0,0 +1,587 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Spark 3.5: Spark property for avoiding write failures due to missing fields in complex data types [iceberg]

2024-06-13 Thread via GitHub
aleenamg21-1 commented on PR #10361: URL: https://github.com/apache/iceberg/pull/10361#issuecomment-2167168805 @nastra Please review this P.R -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] How to avoid partition key sorting when inserting data into a partitioned Iceberg table? [iceberg]

2024-06-13 Thread via GitHub
eubnara commented on issue #10181: URL: https://github.com/apache/iceberg/issues/10181#issuecomment-2167143637 How about this? I don't know the reason why but if I add `dt != NULL` condition, `SORT` disappered on `explain`. ``` insert into temp.partition_table select dt

Re: [I] How to avoid partition key sorting when inserting data into a partitioned Iceberg table? [iceberg]

2024-06-13 Thread via GitHub
eubnara commented on issue #10181: URL: https://github.com/apache/iceberg/issues/10181#issuecomment-2167141423 How about try? I don't know the reason why ``` insert into temp.partition_table select dt ,contents from temp.dataset where dt = '20240418' and dt !

[I] No module named 'pyiceberg.table.partitioning' [iceberg]

2024-06-13 Thread via GitHub
mounahaadi opened a new issue, #10491: URL: https://github.com/apache/iceberg/issues/10491 When I run from pyiceberg.table.partitioning import PartitionSpec, PartitionField from pyiceberg.transforms import DayTransform Getting error like this -

[I] BUG: Bug: partition name stored in partition data in data file contains special character [iceberg-python]

2024-06-13 Thread via GitHub
puchengy opened a new issue, #175: URL: https://github.com/apache/iceberg-python/issues/175 ### Apache Iceberg version 0.5.0 (latest release) ### Please describe the bug 🐞 an example to demonstrate the issue https://github.com/puchengy/iceberg-python/commit/6bd4cefe74aae

Re: [I] BUG: Bug: partition name stored in partition data in data file contains special character [iceberg-python]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #175: URL: https://github.com/apache/iceberg-python/issues/175#issuecomment-2166989097 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] BUG: Bug: partition name stored in partition data in data file contains special character [iceberg-python]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #175: BUG: Bug: partition name stored in partition data in data file contains special character URL: https://github.com/apache/iceberg-python/issues/175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] Flink: The primary key name of the same iceberg table should be same during different queries [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2644: URL: https://github.com/apache/iceberg/issues/2644#issuecomment-2166987588 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: The primary key name of the same iceberg table should be same during different queries [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2644: Flink: The primary key name of the same iceberg table should be same during different queries URL: https://github.com/apache/iceberg/issues/2644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Bug in emitted ScanEvent in BaseAllMetadataTableScan [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2635: Bug in emitted ScanEvent in BaseAllMetadataTableScan URL: https://github.com/apache/iceberg/issues/2635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] UnsupportedOperationException: Byte-buffer read unsupported by input stream [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2631: UnsupportedOperationException: Byte-buffer read unsupported by input stream URL: https://github.com/apache/iceberg/issues/2631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] UnsupportedOperationException: Byte-buffer read unsupported by input stream [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2631: URL: https://github.com/apache/iceberg/issues/2631#issuecomment-2166987552 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] trino iceberg planning time so long [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2619: trino iceberg planning time so long URL: https://github.com/apache/iceberg/issues/2619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] trino iceberg planning time so long [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2619: URL: https://github.com/apache/iceberg/issues/2619#issuecomment-2166987529 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink CDC iceberg table have duplicate rows [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2610: URL: https://github.com/apache/iceberg/issues/2610#issuecomment-2166987514 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Bug in emitted ScanEvent in BaseAllMetadataTableScan [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2635: URL: https://github.com/apache/iceberg/issues/2635#issuecomment-2166987574 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink CDC iceberg table have duplicate rows [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2610: Flink CDC iceberg table have duplicate rows URL: https://github.com/apache/iceberg/issues/2610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Feature Request: To query iceberg tables from BI tools(like tableau,DBVisualizer) using jdbc/odbc connectors [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2605: URL: https://github.com/apache/iceberg/issues/2605#issuecomment-2166987491 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Feature Request: To query iceberg tables from BI tools(like tableau,DBVisualizer) using jdbc/odbc connectors [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2605: Feature Request: To query iceberg tables from BI tools(like tableau,DBVisualizer) using jdbc/odbc connectors URL: https://github.com/apache/iceberg/issues/2605 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [I] Hive aggregate query iceberg tables is failing with ArrayIndexOutOfBound exception using Hive Catalog [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2601: URL: https://github.com/apache/iceberg/issues/2601#issuecomment-2166987481 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Hive aggregate query iceberg tables is failing with ArrayIndexOutOfBound exception using Hive Catalog [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2601: Hive aggregate query iceberg tables is failing with ArrayIndexOutOfBound exception using Hive Catalog URL: https://github.com/apache/iceberg/issues/2601 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Allow Type Promotion to String [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2594: URL: https://github.com/apache/iceberg/issues/2594#issuecomment-2166987464 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Partition Spec Evolution [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2589: URL: https://github.com/apache/iceberg/issues/2589#issuecomment-2166987451 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Allow Type Promotion to String [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2594: Allow Type Promotion to String URL: https://github.com/apache/iceberg/issues/2594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Allow SchemaToType class to be public so we can extends and override `record` method [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2580: URL: https://github.com/apache/iceberg/issues/2580#issuecomment-2166987422 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Partition Spec Evolution [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2589: Partition Spec Evolution URL: https://github.com/apache/iceberg/issues/2589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] Allow SchemaToType class to be public so we can extends and override `record` method [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2580: Allow SchemaToType class to be public so we can extends and override `record` method URL: https://github.com/apache/iceberg/issues/2580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Add geometry type to iceberg [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2586: URL: https://github.com/apache/iceberg/issues/2586#issuecomment-2166987432 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Cannot unset hive.engine.enabled once enabled if Iceberg Jars are not on the HMS Classpath [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] commented on issue #2578: URL: https://github.com/apache/iceberg/issues/2578#issuecomment-2166987411 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Cannot unset hive.engine.enabled once enabled if Iceberg Jars are not on the HMS Classpath [iceberg]

2024-06-13 Thread via GitHub
github-actions[bot] closed issue #2578: Cannot unset hive.engine.enabled once enabled if Iceberg Jars are not on the HMS Classpath URL: https://github.com/apache/iceberg/issues/2578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Bump pydantic from 2.7.3 to 2.7.4 [iceberg-python]

2024-06-13 Thread via GitHub
dependabot[bot] opened a new pull request, #816: URL: https://github.com/apache/iceberg-python/pull/816 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.7.3 to 2.7.4. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

[PR] Bump mypy-boto3-glue from 1.34.121 to 1.34.126 [iceberg-python]

2024-06-13 Thread via GitHub
dependabot[bot] opened a new pull request, #815: URL: https://github.com/apache/iceberg-python/pull/815 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.121 to 1.34.126. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commi

[PR] Bump pypa/cibuildwheel from 2.19.0 to 2.19.1 [iceberg-python]

2024-06-13 Thread via GitHub
dependabot[bot] opened a new pull request, #814: URL: https://github.com/apache/iceberg-python/pull/814 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.19.0 to 2.19.1. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwhee

Re: [PR] Run Flink, Spark3 tests on Java 17 too [iceberg]

2024-06-13 Thread via GitHub
stevenzwu commented on PR #10477: URL: https://github.com/apache/iceberg/pull/10477#issuecomment-2166898276 > Of course, who knows, maybe the users never use the Iceberg Spark or Iceberg Flink code with e.g. Java 17 and therefore we don't need to support that -- is this the case? I k

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638672069 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638494238 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add missing arrow predicate pushdown implementations for `StartsWith`, `NotStartsWith`, `In`, and `NotIn` [iceberg-rust]

2024-06-13 Thread via GitHub
sdd commented on PR #404: URL: https://github.com/apache/iceberg-rust/pull/404#issuecomment-2166791460 Lol, of all the possible names I could have chosen for a new column, I chose the one name that was an exact match for the column that shouldn't exist in the `test_select_no_exist_column` t

Re: [PR] Support building with Java 21 [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on PR #10474: URL: https://github.com/apache/iceberg/pull/10474#issuecomment-2166466182 others here are more familiar with the build than I am, will let them comment. Is this removing some java version? Also, why all the code formatting, can we remove those from t

Re: [I] Implement all functions of BoundPredicateVisitor for ManifestFilterVisitor [iceberg-rust]

2024-06-13 Thread via GitHub
s-akhtar-baig closed issue #350: Implement all functions of BoundPredicateVisitor for ManifestFilterVisitor URL: https://github.com/apache/iceberg-rust/issues/350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[PR] Add missing arrow predicate pushdown implementations for `StartsWith`, `NotStartsWith`, `In`, and `NotIn` [iceberg-rust]

2024-06-13 Thread via GitHub
sdd opened a new pull request, #404: URL: https://github.com/apache/iceberg-rust/pull/404 Resolves 4 outstanding TODOs in the Arrow Reader so that we can do proper predicate pushdown for all of the remaining operators that were not yet implemented. -- This is an automated message from th

Re: [PR] Run Flink, Spark3 tests on Java 17 too [iceberg]

2024-06-13 Thread via GitHub
findepi commented on code in PR #10477: URL: https://github.com/apache/iceberg/pull/10477#discussion_r1638758353 ## .github/workflows/spark-ci.yml: ## @@ -69,7 +69,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: -jvm: [8, 11] +jvm: [8, 11, 17]

Re: [PR] Support building with Java 21 [iceberg]

2024-06-13 Thread via GitHub
findepi commented on PR #10474: URL: https://github.com/apache/iceberg/pull/10474#issuecomment-2166580206 > others here are more familiar with the build than I am, will let them comment. Is this removing some java version? no, i don't think so > Also, why all the code formattin

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-13 Thread via GitHub
Fokko merged PR #367: URL: https://github.com/apache/iceberg-rust/pull/367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Iceberg does not work with Spark's default hive metastore (embedded Derby database) [iceberg]

2024-06-13 Thread via GitHub
ezdine commented on issue #7847: URL: https://github.com/apache/iceberg/issues/7847#issuecomment-2166509977 I am trying to do something similar where the catalog is a remote rest catalog. But getting the same error. Any help would be appreciated. ``` spark-sql --packages org.apache.ice

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638672069 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638468180 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638494238 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1628585371 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638494238 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638458790 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638458790 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638446379 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-13 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1638446379 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Support building with Java 21 [iceberg]

2024-06-13 Thread via GitHub
jbonofre commented on PR #10474: URL: https://github.com/apache/iceberg/pull/10474#issuecomment-2165897342 @findepi thanks for the PR. As you know, I'm working on revapi and gradle update. Supporting JDK21 is great, but we might need to keep previous Java version for some engine/plug

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
ZENOTME commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2165826003 The drawback of the current implementation is containing the metadata in every scan task may be a cost. One solution is to provide an abstract to group the task to share the metad

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 merged PR #400: URL: https://github.com/apache/iceberg-rust/pull/400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] SnapshotRetention::Tag max_ref_age_ms should be optional [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 merged PR #391: URL: https://github.com/apache/iceberg-rust/pull/391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] bug: SnapshotRetention::Tag max_ref_age_ms should be optional [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 closed issue #392: bug: SnapshotRetention::Tag max_ref_age_ms should be optional URL: https://github.com/apache/iceberg-rust/issues/392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1638297920 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,264 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { refer

Re: [I] bug: Iceberg opendal missing s3 support [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 closed issue #385: bug: Iceberg opendal missing s3 support URL: https://github.com/apache/iceberg-rust/issues/385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] field-id in avro schema is missing [iceberg-rust]

2024-06-13 Thread via GitHub
ZENOTME closed issue #131: field-id in avro schema is missing URL: https://github.com/apache/iceberg-rust/issues/131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-13 Thread via GitHub
Fokko commented on code in PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#discussion_r1638292445 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -103,98 +106,264 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { reference: &B

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
Xuanwo commented on code in PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#discussion_r1638278859 ## crates/catalog/glue/Cargo.toml: ## @@ -33,7 +33,7 @@ anyhow = { workspace = true } async-trait = { workspace = true } aws-config = { workspace = true } aws-sdk-

Re: [PR] Run Flink, Spark3 tests on Java 17 too [iceberg]

2024-06-13 Thread via GitHub
singhpk234 commented on code in PR #10477: URL: https://github.com/apache/iceberg/pull/10477#discussion_r1638279561 ## .github/workflows/spark-ci.yml: ## @@ -69,7 +69,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: -jvm: [8, 11] +jvm: [8, 11,

[I] Custom s3 endpoint: Unable to execute HTTP request: Remote host terminated the handshake [iceberg]

2024-06-13 Thread via GitHub
samueljackson92 opened a new issue, #10490: URL: https://github.com/apache/iceberg/issues/10490 ### Apache Iceberg version 1.5.2 (latest release) ### Query engine None ### Please describe the bug 🐞 Hi, I am experimenting with setting up Iceberg locall

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on code in PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#discussion_r1638266798 ## crates/catalog/glue/Cargo.toml: ## @@ -33,7 +33,7 @@ anyhow = { workspace = true } async-trait = { workspace = true } aws-config = { workspace = true } a

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2165789205 Thanks for the explaination, I'll take a look at the pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] bug: SnapshotRetention::Tag max_ref_age_ms should be optional [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on issue #392: URL: https://github.com/apache/iceberg-rust/issues/392#issuecomment-2165778976 Close by #391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
ZENOTME commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2165768384 > Hi, @ZENOTME I think already there exists a `to_arrow` method here: > > https://github.com/apache/iceberg-rust/blob/15e61f23198c4cc5d320d631e22e2fbc02d167c8/crates/iceberg

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2165707437 Hi, @ZENOTME I think already there exists a `to_arrow` method here: https://github.com/apache/iceberg-rust/blob/15e61f23198c4cc5d320d631e22e2fbc02d167c8/crates/iceberg/src/s

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
Xuanwo commented on code in PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#discussion_r1638234228 ## crates/iceberg/Cargo.toml: ## @@ -28,6 +28,10 @@ repository = { workspace = true } license = { workspace = true } keywords = ["iceberg"] +[features] +storage-f

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
Xuanwo commented on PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#issuecomment-2165725815 cc @liurenjie1024, please review again, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on PR #401: URL: https://github.com/apache/iceberg-rust/pull/401#issuecomment-2165718010 Sorry for late reply, let's discuss here:https://github.com/apache/iceberg-rust/issues/398 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on issue #398: URL: https://github.com/apache/iceberg-rust/issues/398#issuecomment-2165711870 But I agree that we should not make the `ArrowReaderBuilder` 's methods crate private rather a public api since it's error prone. -- This is an automated message from the

Re: [PR] Implement BoundPredicateVisitor trait for ManifestFilterVisitor [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on PR #367: URL: https://github.com/apache/iceberg-rust/pull/367#issuecomment-2165695873 cc @Fokko Do you want to take a review for this? I see you assigned yourself for review. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Spark: Dropping partition column from old partition table corrupts entire table [iceberg]

2024-06-13 Thread via GitHub
lurnagao-dahua commented on issue #10234: URL: https://github.com/apache/iceberg/issues/10234#issuecomment-2165495423 Hi, may I ask the repair method is to traverse all schemas or the partition spec persist with schema id in metadata? -- This is an automated message from the Apache

Re: [PR] Enhancement: refine the reader interface [iceberg-rust]

2024-06-13 Thread via GitHub
ZENOTME commented on code in PR #401: URL: https://github.com/apache/iceberg-rust/pull/401#discussion_r1638068933 ## crates/iceberg/src/scan.rs: ## @@ -503,17 +544,36 @@ pub struct FileScanTask { start: u64, #[allow(dead_code)] length: u64, +project_field_id:

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on code in PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#discussion_r1638044746 ## crates/iceberg/Cargo.toml: ## @@ -28,6 +28,10 @@ repository = { workspace = true } license = { workspace = true } keywords = ["iceberg"] +[features] +st

Re: [I] Spark on iceberg table is slower than spark on hive parquet table [iceberg]

2024-06-13 Thread via GitHub
smallx closed issue #2562: Spark on iceberg table is slower than spark on hive parquet table URL: https://github.com/apache/iceberg/issues/2562 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Spark on iceberg table is slower than spark on hive parquet table [iceberg]

2024-06-13 Thread via GitHub
smallx commented on issue #2562: URL: https://github.com/apache/iceberg/issues/2562#issuecomment-2165320620 closing this outdated pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] How can I use Iceberg in C++ [iceberg]

2024-06-13 Thread via GitHub
MRocholl commented on issue #5122: URL: https://github.com/apache/iceberg/issues/5122#issuecomment-2165277014 just stumbled across [delta-kernel-rs](https://github.com/delta-incubator/delta-kernel-rs). In my opinion this would also be what iceberg requires. One project to enable queries fr

Re: [PR] Fix `rewrite_position_delete_files` result file set [iceberg]

2024-06-13 Thread via GitHub
bk-mz commented on PR #9945: URL: https://github.com/apache/iceberg/pull/9945#issuecomment-2165151682 This looks plain wrong. If I we set shuffle.partititions to 1 this will affect the rewrite procedure, it will collapse all delete files into one, meaning we'd be having a situation

Re: [PR] feat: Add storage features for iceberg [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 commented on code in PR #400: URL: https://github.com/apache/iceberg-rust/pull/400#discussion_r1637923809 ## crates/iceberg/Cargo.toml: ## @@ -28,6 +28,10 @@ repository = { workspace = true } license = { workspace = true } keywords = ["iceberg"] +[features] +st

Re: [PR] Fix `rewrite_position_delete_files` result file set [iceberg]

2024-06-13 Thread via GitHub
bk-mz closed pull request #9945: Fix `rewrite_position_delete_files` result file set URL: https://github.com/apache/iceberg/pull/9945 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Bump arrow related crates to 52 [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 closed issue #399: Bump arrow related crates to 52 URL: https://github.com/apache/iceberg-rust/issues/399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] deps: Bump arrow related crates to 52 [iceberg-rust]

2024-06-13 Thread via GitHub
liurenjie1024 merged PR #403: URL: https://github.com/apache/iceberg-rust/pull/403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Bump arrow related crates to 52 [iceberg-rust]

2024-06-13 Thread via GitHub
Xuanwo commented on issue #399: URL: https://github.com/apache/iceberg-rust/issues/399#issuecomment-2165014905 > Maybe I can try to handle this? @Xuanwo Great, thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-13 Thread via GitHub
HonahX commented on code in PR #807: URL: https://github.com/apache/iceberg-python/pull/807#discussion_r1637685511 ## pyiceberg/io/pyarrow.py: ## @@ -1167,8 +1197,14 @@ def __init__(self, file_schema: Schema): def _cast_if_needed(self, field: NestedField, values: pa.Array

Re: [PR] Flink 1.17: Supports batch queries using time ranges [iceberg]

2024-06-13 Thread via GitHub
hililiwei commented on PR #7362: URL: https://github.com/apache/iceberg/pull/7362#issuecomment-2164929089 > Welcome back @hililiwei! Went through the PR and left a few comments. Would you mind checking the test failures? Thank you @pvary . Over the past year, I have been intermittentl

Re: [PR] Flink 1.17: Support Partition Commit notification [iceberg]

2024-06-13 Thread via GitHub
hililiwei closed pull request #7638: Flink 1.17: Support Partition Commit notification URL: https://github.com/apache/iceberg/pull/7638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

  1   2   >