Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

2024-05-05 Thread via GitHub
a-agmon commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2095268461 Thanks, @Fokko and @zeodtr, for the clarifications and explanations! I think that it's important to fix this in the next release, as the current situation is that the Rust API

Re: [I] Add runtime module to enable concurrent load of manifest files. [iceberg-rust]

2024-05-05 Thread via GitHub
liurenjie1024 commented on issue #124: URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2095266262 Hi, @marvinlanhenke After #233 got merged, we will have a basic runtime framework. > Have you already made up your mind; Not yet. I think you solution g

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-05-05 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1590563280 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +219,634 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-05-05 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1590563196 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +219,634 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-05-05 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1590553330 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +219,634 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [I] Add Apache Iceberg connector/support for PostgreSQL [iceberg]

2024-05-05 Thread via GitHub
CincyBC commented on issue #8279: URL: https://github.com/apache/iceberg/issues/8279#issuecomment-2095088061 @vdittgen There isn't a Foreign Data Wrapper for Iceberg at this time, but you can query from PSQL using PyIceberg. See this writeup for how: https://medium.com/datareply/unlocking-t

Re: [PR] Test out Parquet 1.14.0 [iceberg]

2024-05-05 Thread via GitHub
Fokko commented on PR #10209: URL: https://github.com/apache/iceberg/pull/10209#issuecomment-2094939429 The tests are failing, but that's because Jackson ships with JDK21-specific code, and the shadow plugin cannot cope with that. There is a fork of the plugin, but that one requires Gradle

Re: [PR] Hive catalog: Add retry logic for hive locking [iceberg-python]

2024-05-05 Thread via GitHub
kevinjqliu commented on code in PR #701: URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1590382950 ## pyiceberg/catalog/hive.py: ## @@ -111,6 +122,13 @@ HIVE2_COMPATIBLE = "hive.hive2-compatible" HIVE2_COMPATIBLE_DEFAULT = False +DEFAULT_LOCK_CHECK_MIN_WA

Re: [I] ValueError: Mismatch in fields: ? [iceberg-python]

2024-05-05 Thread via GitHub
kevinjqliu commented on issue #674: URL: https://github.com/apache/iceberg-python/issues/674#issuecomment-2094904451 Oh btw, a workaround before the fix is merged is to reorder the schemas to match manually :) Can sort by `field_id`, similar to the fix -- This is an automated message

[PR] Remove trailing slash from table location when creating a table [iceberg-python]

2024-05-05 Thread via GitHub
felixscherz opened a new pull request, #702: URL: https://github.com/apache/iceberg-python/pull/702 Hi, this is intended to resolve https://github.com/apache/iceberg-python/issues/606. Changes: - use `rstrip("/")` when a location is specified for `create_table` - added tests for

Re: [I] Can not parse sort order when there are more than one sql extensions specified and iceberg extension is not placed at the right most position [iceberg]

2024-05-05 Thread via GitHub
ajantha-bhat commented on issue #8004: URL: https://github.com/apache/iceberg/issues/8004#issuecomment-2094841965 Yeah, this problem is only for sort order parsing of REWRITE call procedure because Iceberg uses some custom parsing code to parse sort order as of now. From spark, parse

[PR] Hive catalog: Add retry logic for hive locking [iceberg-python]

2024-05-05 Thread via GitHub
frankliee opened a new pull request, #701: URL: https://github.com/apache/iceberg-python/pull/701 In the current hive catalog implementation, locking will only be executed once. If another task has already locked the target table, the locking will fail inevitably. This PR adds ret

Re: [I] Failing to create a table using pyiceberg [iceberg-python]

2024-05-05 Thread via GitHub
salexln closed issue #692: Failing to create a table using pyiceberg URL: https://github.com/apache/iceberg-python/issues/692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Failing to create a table using pyiceberg [iceberg-python]

2024-05-05 Thread via GitHub
salexln commented on issue #692: URL: https://github.com/apache/iceberg-python/issues/692#issuecomment-2094764517 thanks for all the helpers! eventually what worked is to set the correct credentials in `~/.aws/credentials` and then ```python catalog = load_catalog("glue", **{"ty

Re: [PR] Add `InclusiveMetricsEvaluator` [iceberg-rust]

2024-05-05 Thread via GitHub
sdd commented on code in PR #347: URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590269988 ## crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs: ## @@ -0,0 +1,2255 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [PR] Add `InclusiveMetricsEvaluator` [iceberg-rust]

2024-05-05 Thread via GitHub
sdd commented on code in PR #347: URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590269988 ## crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs: ## @@ -0,0 +1,2255 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [PR] Add `InclusiveMetricsEvaluator` [iceberg-rust]

2024-05-05 Thread via GitHub
sdd commented on code in PR #347: URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590269295 ## crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs: ## @@ -0,0 +1,2255 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [PR] Add `InclusiveMetricsEvaluator` [iceberg-rust]

2024-05-05 Thread via GitHub
marvinlanhenke commented on code in PR #347: URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590269281 ## crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs: ## @@ -0,0 +1,2255 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// o

Re: [PR] Add `InclusiveMetricsEvaluator` [iceberg-rust]

2024-05-05 Thread via GitHub
sdd commented on code in PR #347: URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590268796 ## crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs: ## @@ -0,0 +1,2255 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [PR] Build: Bump mkdocs-material from 9.5.19 to 9.5.21 [iceberg]

2024-05-05 Thread via GitHub
Fokko merged PR #10272: URL: https://github.com/apache/iceberg/pull/10272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] define ray datasource for Iceberg tables [iceberg-python]

2024-05-05 Thread via GitHub
prabodh1194 commented on PR #698: URL: https://github.com/apache/iceberg-python/pull/698#issuecomment-2094673672 closing in lieu of the mentioned PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] define ray datasource for Iceberg tables [iceberg-python]

2024-05-05 Thread via GitHub
prabodh1194 closed pull request #698: define ray datasource for Iceberg tables URL: https://github.com/apache/iceberg-python/pull/698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm