Re: [I] reset catalog can't reslove s3 domain name [iceberg]

2024-03-19 Thread via GitHub
ming12713 commented on issue #7709: URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2008705570 > > Is there anyone who solved this issue? > > Try to add env variable to the rest container: `CATALOG_S3_PATH__STYLE__ACCESS: true` > > It'll be [converted](https://

Re: [PR] Migrate Transactions, Partitions and Statistics files in Core to JUnit5 [iceberg]

2024-03-19 Thread via GitHub
tomtongue commented on PR #: URL: https://github.com/apache/iceberg/pull/#issuecomment-2008692723 @nastra Thanks for the review. Could you check the new one when you have time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-19 Thread via GitHub
a-agmon commented on code in PR #284: URL: https://github.com/apache/iceberg-rust/pull/284#discussion_r1531518572 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1052,6 +1052,89 @@ pub struct DataFile { pub(crate) sort_order_id: Option, } +impl DataFile { +/// Get the

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-03-19 Thread via GitHub
matepek commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2008666705 I see what you meant now.. `spark.sql.catalog.spark_catalog.type` was configured to `jdbc` which was actually a mistake of mine. But not defining the `spark_catalog`

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008661653 i just want to confirm while i use spark with iceberg catalog "org.apache.iceberg.spark.SparkCatalog" and HMS, if it's normal or not when it can't delete table directories after

Re: [I] reset catalog can't reslove s3 domain name [iceberg]

2024-03-19 Thread via GitHub
dongsupkim-onepredict commented on issue #7709: URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2008658279 It works Thanks! @sg-c0de -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008655496 > alter table from external table to managed table This only works with Hive CLI. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-03-19 Thread via GitHub
matepek commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2008652290 What do you mean by that I'm using JDBC catalog? I thought `spark.sql.catalogImplementation = hive` sets it to hive catalog. (I know I have a knowledge gap and I'm trying to

Re: [PR] feat: make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
himadripal commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531493828 ## crates/catalog/rest/src/catalog.rs: ## @@ -134,6 +134,22 @@ impl RestCatalogConfig { Client::builder().default_headers(headers).build()?,

Re: [PR] feat: make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
himadripal commented on PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#issuecomment-2008642194 Thank you @flyrain and @liurenjie1024 for reviewing, I have fixed all the comments. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008641881 and i have checked other comments on this kind issue , why can't delete table directories when drop table , it's because the table is a external table when created by spark, and givi

Re: [PR] Glue endpoint config variable [iceberg-python]

2024-03-19 Thread via GitHub
HonahX commented on PR #530: URL: https://github.com/apache/iceberg-python/pull/530#issuecomment-2008641566 @sebpretzer FYI, I just found an error in the current glue integration test: #536 . Please feel free to include this fix in this PR if you want to add `glue.endpoint` to the integrati

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008639451 HMS, if i use HMS catalog , i can't delete table directories when i drop table, even i drop table with purge. the software version is: spark 3.3.2, hive 2.3.9, iceberg jar is:1.4.0.

Re: [PR] feat: make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
himadripal commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531494082 ## crates/catalog/rest/src/catalog.rs: ## @@ -886,6 +903,51 @@ mod tests { config_mock.assert_async().await; } +#[tokio::test] +async fn t

Re: [PR] feat: make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
himadripal commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531493828 ## crates/catalog/rest/src/catalog.rs: ## @@ -134,6 +134,22 @@ impl RestCatalogConfig { Client::builder().default_headers(headers).build()?,

[I] [feat request] Make `Table` / `TableMetadata` JSON serializable [iceberg-python]

2024-03-19 Thread via GitHub
kevinjqliu opened a new issue, #535: URL: https://github.com/apache/iceberg-python/issues/535 ### Feature Request / Improvement The REST Catalog exposes `Table` and `TableMetadata` information as HTTP endpoints in JSON format ([link](https://iceberg-rest-image.fly.dev/v1/namespaces/d

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
stevenzwu commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1531487236 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,21 @@ public void flush() throws IOException { @Override public void

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1531467885 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1531467885 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008591571 Which catalogs do you use in those two cases? Can you share the configs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1531450892 ## crates/iceberg/src/arrow.rs: ## @@ -106,3 +114,732 @@ impl ArrowReader { ProjectionMask::all() } } + +/// A post order arrow schema visitor.

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-19 Thread via GitHub
viirya commented on PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#issuecomment-2008556920 @liurenjie1024 Thanks for providing some references to #251, #252. I took at the Python reading projection in https://github.com/apache/iceberg-python/blob/6c8ea0effac0942ad4e880e

Re: [PR] feat: Complete predicate builders for all operators. [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #276: URL: https://github.com/apache/iceberg-rust/pull/276#discussion_r1529565534 ## crates/iceberg/src/expr/predicate.rs: ## @@ -282,15 +282,18 @@ impl Bind for Predicate { return Ok(BoundPredicate::AlwaysTrue);

Re: [PR] make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531421328 ## crates/catalog/rest/src/catalog.rs: ## @@ -886,6 +903,51 @@ mod tests { config_mock.assert_async().await; } +#[tokio::test] +async f

Re: [PR] make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531418192 ## crates/catalog/rest/src/catalog.rs: ## @@ -134,6 +134,22 @@ impl RestCatalogConfig { Client::builder().default_headers(headers).build()?,

Re: [PR] make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531417549 ## crates/catalog/rest/src/catalog.rs: ## @@ -134,6 +134,22 @@ impl RestCatalogConfig { Client::builder().default_headers(headers).build()?,

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #284: URL: https://github.com/apache/iceberg-rust/pull/284#discussion_r1531416557 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1052,6 +1052,89 @@ pub struct DataFile { pub(crate) sort_order_id: Option, } +impl DataFile { +/// G

Re: [PR] Glue endpoint config variable [iceberg-python]

2024-03-19 Thread via GitHub
HonahX commented on code in PR #530: URL: https://github.com/apache/iceberg-python/pull/530#discussion_r1531398064 ## tests/catalog/test_glue.py: ## @@ -692,3 +692,11 @@ def test_commit_table_properties( updated_table_metadata = table.metadata assert test_catalog._pars

Re: [I] how to fix org.apache.spark.shuffle.FetchFailedException: [iceberg]

2024-03-19 Thread via GitHub
github-actions[bot] commented on issue #2211: URL: https://github.com/apache/iceberg/issues/2211#issuecomment-2008379192 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1531273698 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1531273698 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
flyrain commented on code in PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#discussion_r1531236117 ## crates/catalog/rest/src/catalog.rs: ## @@ -134,6 +134,22 @@ impl RestCatalogConfig { Client::builder().default_headers(headers).build()?, ))

Re: [PR] make optional oauth param configurable [iceberg-rust]

2024-03-19 Thread via GitHub
himadripal commented on PR #278: URL: https://github.com/apache/iceberg-rust/pull/278#issuecomment-2008034249 @liurenjie1024 @TennyZhuang @flyrain Please take a look. This PR is to make all optional params for oauth configurable. -- This is an automated message from the Apache Git Servi

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-19 Thread via GitHub
a-agmon commented on PR #284: URL: https://github.com/apache/iceberg-rust/pull/284#issuecomment-2007989888 Hi @liurenjie1024 , @Xuanwo I have addressed the issues and updated the PR. Please let me know what do you think -- This is an automated message from the Apache Git Service

Re: [I] Convert row filter to arrow filter [iceberg-rust]

2024-03-19 Thread via GitHub
viirya commented on issue #265: URL: https://github.com/apache/iceberg-rust/issues/265#issuecomment-2007930647 I'm looking into this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530925615 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530923451 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [I] Iceberg does not work with Spark's default hive metastore (embedded Derby database) [iceberg]

2024-03-19 Thread via GitHub
nsucheendran commented on issue #7847: URL: https://github.com/apache/iceberg/issues/7847#issuecomment-2007824814 > I used a hadoop catalog for my testing. Worked fine! Thanks @RussellSpitzer . Hi @vinitamaloo-asu My team is facing the same issue. Is there any code you can share o

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530843897 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [I] SHOW TABLES IN ... returns error but table exists [iceberg]

2024-03-19 Thread via GitHub
nastra commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2007780751 I think the problem is actually the expectation of the `SparkSessionCatalog` behavior and that the documentation doesn't do a good job of describing what it's purpose is (I've stumb

Re: [I] SHOW TABLES IN ... returns error but table exists [iceberg]

2024-03-19 Thread via GitHub
nastra commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2007764290 > It's great that 1.5 supports views. We have been waiting for this for long. But it bleed from some issues. There isn't anything view-related in your steps above. It seems to

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-19 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-2007748345 @jackye1995 @nastra any comments or thoughts on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[I] SHOW TABLES IN ... returns error but table exists [iceberg]

2024-03-19 Thread via GitHub
matepek opened a new issue, #10003: URL: https://github.com/apache/iceberg/issues/10003 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 # The issue It's great that 1.5 supports views. We have bee

Re: [PR] Migrate Transactions, Partitions and Statistics files in Core to JUnit5 [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #: URL: https://github.com/apache/iceberg/pull/#discussion_r1530789489 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -68,38 +66,38 @@ public void testSpecIsUnpartitionedForVoidTranforms() { PartitionSpec sp

Re: [PR] Migrate Transactions, Partitions and Statistics files in Core to JUnit5 [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #: URL: https://github.com/apache/iceberg/pull/#discussion_r1530772051 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -68,38 +66,38 @@ public void testSpecIsUnpartitionedForVoidTranforms() { PartitionSpec sp

Re: [PR] Migrate Transactions, Partitions and Statistics files in Core to JUnit5 [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #: URL: https://github.com/apache/iceberg/pull/#discussion_r1530771482 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -68,38 +66,38 @@ public void testSpecIsUnpartitionedForVoidTranforms() { PartitionSpec sp

Re: [PR] Migrate Transactions, Partitions and Statistics files in Core to JUnit5 [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #: URL: https://github.com/apache/iceberg/pull/#discussion_r1530770150 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -68,38 +66,38 @@ public void testSpecIsUnpartitionedForVoidTranforms() { PartitionSpec sp

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
stevenzwu commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530759291 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that doesn't exist [iceberg]

2024-03-19 Thread via GitHub
stevenzwu commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530758538 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void

Re: [I] Support Nessie catalog [iceberg-python]

2024-03-19 Thread via GitHub
RobPrat commented on issue #19: URL: https://github.com/apache/iceberg-python/issues/19#issuecomment-2007643836 I would also like to know if it is estimated to be worked on soon, I'd find it very useful. Thx! -- This is an automated message from the Apache Git Service. To respond to the m

[PR] Fix dependency with `deptry` [iceberg-python]

2024-03-19 Thread via GitHub
kevinjqliu opened a new pull request, #534: URL: https://github.com/apache/iceberg-python/pull/534 Resolve dependency errors found by `deptry` (added in #528). From 27 -> 17 errors * `botocore` (4 errors) * `typing_extensions` (5 errors) * `pydantic_core` (1 error) De

Re: [PR] AWS, Core: Replace .withFailMessage() usage with .as() [iceberg]

2024-03-19 Thread via GitHub
nastra merged PR #1: URL: https://github.com/apache/iceberg/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-03-19 Thread via GitHub
viirya commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1530679117 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,310 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-19 Thread via GitHub
amogh-jahagirdar commented on PR #9953: URL: https://github.com/apache/iceberg/pull/9953#issuecomment-2007499715 @ajantha-bhat Replied my thoughts on the Trino PR and I'll summarize what I think here: To be clear, there is no extra I/O being done in practice for `newInputfile(DataFi

Re: [PR] `add_files` support partitioned tables [iceberg-python]

2024-03-19 Thread via GitHub
syun64 commented on PR #531: URL: https://github.com/apache/iceberg-python/pull/531#issuecomment-2007469549 > @syun64 Thanks for working on this, this looks great! Thank you very much for the detailed review @Fokko . I've adopted all of your review comments 👍 - I would appreciate anot

Re: [PR] `add_files` support partitioned tables [iceberg-python]

2024-03-19 Thread via GitHub
syun64 commented on code in PR #531: URL: https://github.com/apache/iceberg-python/pull/531#discussion_r1530595279 ## pyiceberg/io/pyarrow.py: ## @@ -1594,29 +1595,88 @@ def parquet_path_to_id_mapping( return result -def fill_parquet_file_metadata( -data_file: DataF

Re: [PR] [core] fix #9997 - Handle s3a file upload interrupt which results in table metadata pointing to files that don't exists [iceberg]

2024-03-19 Thread via GitHub
abmo-x commented on code in PR #9998: URL: https://github.com/apache/iceberg/pull/9998#discussion_r1530595107 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java: ## @@ -185,8 +185,20 @@ public void flush() throws IOException { @Override public void cl

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530591056 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations { //

Re: [PR] `add_files` support partitioned tables [iceberg-python]

2024-03-19 Thread via GitHub
syun64 commented on code in PR #531: URL: https://github.com/apache/iceberg-python/pull/531#discussion_r1530566151 ## tests/integration/test_add_files.py: ## @@ -238,3 +239,109 @@ def test_add_files_to_unpartitioned_table_with_schema_updates(spark: SparkSessio for col in d

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530563958 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530563958 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530563958 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530503277 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530535837 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530529910 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530529910 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530529910 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530503277 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530503277 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations

Re: [PR] WIP: Incremental Append Scan [iceberg-python]

2024-03-19 Thread via GitHub
hililiwei commented on code in PR #533: URL: https://github.com/apache/iceberg-python/pull/533#discussion_r1530464016 ## pyiceberg/table/__init__.py: ## @@ -1578,6 +1595,120 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arrow())

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10002: URL: https://github.com/apache/iceberg/pull/10002#discussion_r1530436787 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java: ## @@ -66,6 +67,9 @@ class GlueTableOperations extends BaseMetastoreTableOperations { //

Re: [I] DataFrameWriterV2 doesn't create partitions when using partitionedBy [iceberg]

2024-03-19 Thread via GitHub
Yotamho commented on issue #4363: URL: https://github.com/apache/iceberg/issues/4363#issuecomment-2007206534 > You should add a partition column by `withColumn` and call `partitionedBy` with the new column. Will it support hidden partition this way? I think that in this way, when I w

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530379477 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +264,66 @@ public void renameTable(TableIdentifier from, TableIdentifier origi

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530375282 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -169,8 +260,8 @@ default Table newHmsTable(String hmsTableOwner) {

Re: [PR] Change Append/Overwrite API to accept snapshot properties [iceberg-python]

2024-03-19 Thread via GitHub
Fokko commented on PR #419: URL: https://github.com/apache/iceberg-python/pull/419#issuecomment-2007184688 Thanks @Gowthami03B for adding this 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Change Append/Overwrite API to accept snapshot properties [iceberg-python]

2024-03-19 Thread via GitHub
Fokko merged PR #419: URL: https://github.com/apache/iceberg-python/pull/419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] `add_files` support partitioned tables [iceberg-python]

2024-03-19 Thread via GitHub
syun64 commented on code in PR #531: URL: https://github.com/apache/iceberg-python/pull/531#discussion_r1530323834 ## pyiceberg/io/pyarrow.py: ## @@ -1594,29 +1595,88 @@ def parquet_path_to_id_mapping( return result -def fill_parquet_file_metadata( -data_file: DataF

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nk1506 commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530321209 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -309,65 +304,39 @@ protected enum CommitStatus { * @return Commit Status of Succe

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nk1506 commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530317976 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -169,8 +260,8 @@ default Table newHmsTable(String hmsTableOwner) {

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-03-19 Thread via GitHub
anupam-saini commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1527602013 ## pyiceberg/catalog/__init__.py: ## @@ -710,6 +760,45 @@ def _get_updated_props_and_update_summary( return properties_update_summary, updated_prop

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1530195937 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,42 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Incremental Append Scan [iceberg-python]

2024-03-19 Thread via GitHub
hililiwei commented on code in PR #533: URL: https://github.com/apache/iceberg-python/pull/533#discussion_r1530185804 ## pyiceberg/table/__init__.py: ## @@ -3014,3 +3145,35 @@ def _new_field_id(self) -> int: def _is_duplicate_partition(self, transform: Transform[Any, Any]

Re: [PR] Incremental Append Scan [iceberg-python]

2024-03-19 Thread via GitHub
hililiwei commented on code in PR #533: URL: https://github.com/apache/iceberg-python/pull/533#discussion_r1530184252 ## pyiceberg/table/__init__.py: ## @@ -1578,6 +1595,120 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arrow())

[PR] Incremental Append Scan [iceberg-python]

2024-03-19 Thread via GitHub
hililiwei opened a new pull request, #533: URL: https://github.com/apache/iceberg-python/pull/533 Hi,@Fokko, long time no see. I have written some preliminary code for incremental reading, which still has many areas for improvement. However, I would like to discuss it with you at an early

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-19 Thread via GitHub
nk1506 commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1530162549 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -275,12 +275,7 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530154379 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -169,8 +260,8 @@ default Table newHmsTable(String hmsTableOwner) {

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530152090 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +264,66 @@ public void renameTable(TableIdentifier from, TableIdentifier origi

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530149331 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -222,24 +214,46 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530146675 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -222,24 +214,46 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530145798 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -222,24 +214,46 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530141957 ## core/src/main/java/org/apache/iceberg/CatalogUtil.java: ## @@ -136,6 +138,18 @@ public static void dropTableData(FileIO io, TableMetadata metadata) { deleteFi

Re: [PR] Hive: Arrange common part of the code for Iceberg View. [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #10001: URL: https://github.com/apache/iceberg/pull/10001#discussion_r1530138582 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -309,65 +304,39 @@ protected enum CommitStatus { * @return Commit Status of Succe

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2024-03-19 Thread via GitHub
nastra commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2006792641 @carlosescura the issue itself hasn't be solved yet. I'm not sure if @rahil-c is actively working on this issue. If not, maybe someone else from the community is interested in working on t

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1530123440 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,25 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-19 Thread via GitHub
nastra commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1530121449 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -275,12 +275,7 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2024-03-19 Thread via GitHub
carlosescura commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2006763129 @lokeshrdy Still doesn't work using Spark `3.5.0`and Iceberg `1.5.0` and Glue as catalog with the following config: ``` SPARK_SETTINGS = [ ( "spark.jars",

[PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-03-19 Thread via GitHub
stephen-shelby opened a new pull request, #10002: URL: https://github.com/apache/iceberg/pull/10002 For all Iceberg table operations (select, insert, alter, etc..) Glue tries to load the Iceberg table by reading in its metadata files. When loading an iceberg table and aws properties are i

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-19 Thread via GitHub
a-agmon commented on PR #284: URL: https://github.com/apache/iceberg-rust/pull/284#issuecomment-2006678166 > Hi @a-agmon Thanks for this pr. But per our previous discussion, we prefer to expose getter methods rather fields directly to user, you can see [the discussion](https://github.com/ap

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-19 Thread via GitHub
marvinlanhenke commented on PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#issuecomment-2006672626 > > yes you're correct - but I think the difference is when we checked `properties.is_emtpy()` and returned `None` the serialization (due to serde config) was completely skipped

Re: [PR] fix: enable public access to ManifestEntry properties [iceberg-rust]

2024-03-19 Thread via GitHub
liurenjie1024 commented on code in PR #284: URL: https://github.com/apache/iceberg-rust/pull/284#discussion_r1530069119 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1049,7 +1049,7 @@ pub struct DataFile { /// order id to null. Readers must ignore sort order id for positio

  1   2   >