Re: [PR] Spark Action to Analyze table [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1593439008 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593438387 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] Support special chars in S3URI [iceberg]

2024-05-07 Thread via GitHub
snazy commented on PR #10283: URL: https://github.com/apache/iceberg/pull/10283#issuecomment-2099816561 > there are cases where they are used As (not so widely) known, special characters in S3 URIs - i.e. those URIs, for example with (back)quotes and hash/question-marks, are not prope

Re: [PR] Spark Action to Analyze table [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1593437419 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593437301 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
jbonofre commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099809671 @ajantha-bhat i was waiting new review from AWS guys. I will move forward by creating a branch with jaxrs/open api annotated interfaces to illustrate the new proposal. -- This

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593432217 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593432217 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099805039 @jbonofre: I don't see much progress in terms os "new proposal" (no response from AWS guys), what are the next plans on that? -- This is an automated message from the Apac

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593429017 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] Spark Action to Analyze table [iceberg]

2024-05-07 Thread via GitHub
karuppayya commented on PR #10288: URL: https://github.com/apache/iceberg/pull/10288#issuecomment-2099800313 cc: @RussellSpitzer @aokolnychyi @huaxingao @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593424223 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593420606 ## format/materialized-view-spec.md: ## @@ -0,0 +1,131 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful too

Re: [PR] Core: FileSystemTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-05-07 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1593419915 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -154,21 +155,33 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593418382 ## format/materialized-view-spec.md: ## @@ -0,0 +1,131 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful too

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593417798 ## format/materialized-view-spec.md: ## @@ -0,0 +1,131 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful too

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593416891 ## format/materialized-view-spec.md: ## @@ -0,0 +1,131 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful too

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593416891 ## format/materialized-view-spec.md: ## @@ -0,0 +1,131 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful too

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

2024-05-07 Thread via GitHub
a-agmon commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2099788232 Added a PR that proposes an interim, but more elegant, solution to the problem. I think. WDYT @Fokko @zeodtr @liurenjie1024 -- This is an automated message from the Apache

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
jbonofre commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099783609 For the context, the `handshake` endpoint of the "new" REST proposal can provide the catalog name, that's the kind of "configuration exchange". I agree that `warehouse` is confusi

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593409234 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergFlinkManifest.java: ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software F

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593408257 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593406692 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundati

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099775221 > I feel like the best way to approach this is to use the warehouse property to identify what HMS catalog name is being addressed It is so confusing to use `warehouse`

[PR] Iceberg.engine.hive.enabled Conf is not honouring for HIVE CATALOG #10286 [iceberg]

2024-05-07 Thread via GitHub
shivjha30 opened a new pull request, #10287: URL: https://github.com/apache/iceberg/pull/10287 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[I] "Iceberg.engine.hive.enabled" Conf is not honouring for HIVE CATALOG [iceberg]

2024-05-07 Thread via GitHub
shivjha30 opened a new issue, #10286: URL: https://github.com/apache/iceberg/issues/10286 ### Apache Iceberg version 1.4.3 ### Query engine Hive ### Please describe the bug ๐Ÿž In the iceberg documentation it could be seen that "To enable Hive support glob

Re: [I] [bug]OversizedAllocationException when query data with Spark [iceberg]

2024-05-07 Thread via GitHub
Xiangakun commented on issue #9820: URL: https://github.com/apache/iceberg/issues/9820#issuecomment-2099696189 Set table property๏ผš`read.parquet.vectorization.batch-size` to a lower value, or just disable VECTORIZATION reader by setting `read.parquet.vectorization.enabled`=false should solve

Re: [PR] Spark 3.4: Incremental scan specify branch [iceberg]

2024-05-07 Thread via GitHub
hililiwei commented on PR #8384: URL: https://github.com/apache/iceberg/pull/8384#issuecomment-2099651239 hi @amogh-jahagirdar, thank you for your review. I have made changes according to the comments. Would you please take a look again? -- This is an automated message from the Ap

[I] Can IceBerg support diskann algorithm ? [iceberg]

2024-05-07 Thread via GitHub
7vw58n3ftvq7cn89v opened a new issue, #10285: URL: https://github.com/apache/iceberg/issues/10285 ### Feature Request / Improvement Diskann is a kind of vector retrieval algo. I didn't find any thing about vector retrieval in Iceberg. ### Query engine None -- Th

Re: [PR] Support special chars in S3URI [iceberg]

2024-05-07 Thread via GitHub
danielcweeks commented on PR #10283: URL: https://github.com/apache/iceberg/pull/10283#issuecomment-2099593480 We shouldn't remove the param/fragment handling because even though they aren't typically used, they are accepted by s3 and there are cases where they are used. This would change

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
szehon-ho commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593277400 ## .palantir/revapi.yml: ## @@ -1018,6 +1018,12 @@ acceptedBreaks: old: "method void org.apache.iceberg.PositionDeletesTable.PositionDeletesBatchScan::(org.

[PR] #10275 - fix NullPointerException [iceberg]

2024-05-07 Thread via GitHub
slessard opened a new pull request, #10284: URL: https://github.com/apache/iceberg/pull/10284 Fix NullPointerException when trying to add the vector's class name to the message for an UnsupportedOperationException. Please see the bug report #10275 for additional context and details

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
szehon-ho commented on PR #10020: URL: https://github.com/apache/iceberg/pull/10020#issuecomment-2099557502 Thanks @RussellSpitzer , addressed initial comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593256062 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593254954 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] docs: Add links checker [iceberg]

2024-05-07 Thread via GitHub
manuzhang commented on PR #9965: URL: https://github.com/apache/iceberg/pull/9965#issuecomment-2099520743 @nastra and @print-sid8 These are known issues for previous versions. I'm a bit surprised it's not fixed in 1.5.1 which was just released. -- This is an automated message from the Apa

[I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]

2024-05-07 Thread via GitHub
cgbur opened a new issue, #716: URL: https://github.com/apache/iceberg-python/issues/716 ### Apache Iceberg version main (development) ### Please describe the bug ๐Ÿž When using the `add_files` table api, the parquet metadata needs to be read and a mapping of `Dict[str, in

[PR] Build: Bump mkdocstrings from 0.25.0 to 0.25.1 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] opened a new pull request, #715: URL: https://github.com/apache/iceberg-python/pull/715 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.25.0 to 0.25.1. Release notes Sourced from https://github.com/mkdocstrings/mkdocstrings/releases";>mkdoc

[PR] Build: Bump tenacity from 8.2.3 to 8.3.0 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] opened a new pull request, #714: URL: https://github.com/apache/iceberg-python/pull/714 Bumps [tenacity](https://github.com/jd/tenacity) from 8.2.3 to 8.3.0. Release notes Sourced from https://github.com/jd/tenacity/releases";>tenacity's releases. 8.3.0 New

[PR] Build: Bump coverage from 7.5.0 to 7.5.1 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] opened a new pull request, #713: URL: https://github.com/apache/iceberg-python/pull/713 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.0 to 7.5.1. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

[PR] Build: Bump sqlalchemy from 2.0.29 to 2.0.30 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] opened a new pull request, #712: URL: https://github.com/apache/iceberg-python/pull/712 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.29 to 2.0.30. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's r

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
flyrain commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099381636 I agreed with @danielcweeks. The Trino client can either pass the catalog name as a part of url(e.g. , `$ENDPOINT/v1/prefix/namespaces/hms_catalog%1Fdb_name/tables/tablename`), or

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
danielcweeks commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099361208 @osscm The information passed to the the REST server as part of the properties is contextual information and is only sent for user sessions. I wouldn't use this for resolving

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593132076 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergFlinkManifest.java: ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Softw

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593131221 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1593129422 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Fou

Re: [I] REST Catalog to support custom-catalog name like HMS/Glue [iceberg]

2024-05-07 Thread via GitHub
osscm commented on issue #10205: URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2099323036 > `iceberg.rest-catalog.warehouse` normally points to a location like `s3://my_bucket/warehouse_location`. Not sure Trino community is OK to use it as the catalog name. Even if that

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593089972 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -140,6 +141,31 @@ public StructType partitionType() { return lazyPartitionType; } +

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593088012 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -140,6 +141,31 @@ public StructType partitionType() { return lazyPartitionType; } +

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593085207 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -140,6 +141,31 @@ public StructType partitionType() { return lazyPartitionType; } +

Re: [I] java.lang.NoClassDefFoundError: scala/jdk/CollectionConverters$ [iceberg]

2024-05-07 Thread via GitHub
nastra commented on issue #10175: URL: https://github.com/apache/iceberg/issues/10175#issuecomment-2099297714 @celltobig are you by any chance running Scala 2.13? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Docs: Add 1.5.2 site docs [iceberg]

2024-05-07 Thread via GitHub
nastra merged PR #10282: URL: https://github.com/apache/iceberg/pull/10282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Docs: Add Javadocs for 1.5.2 [iceberg]

2024-05-07 Thread via GitHub
nastra merged PR #10281: URL: https://github.com/apache/iceberg/pull/10281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Upgrade HadoopTableOperations.version from int32 to long64 [iceberg]

2024-05-07 Thread via GitHub
nastra commented on issue #10277: URL: https://github.com/apache/iceberg/issues/10277#issuecomment-2099282673 @jkolash can you share a little bit more details about the 3rd party that is writing this? It would be good to know why this 3rd party writes this as a long instead of an int. --

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593058949 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -140,6 +141,31 @@ public StructType partitionType() { return lazyPartitionType; } +

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
stevenzwu commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1593047665 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] Core: Calling rewrite_position_delete_files fails on tables with more than 1k columns [iceberg]

2024-05-07 Thread via GitHub
RussellSpitzer commented on code in PR #10020: URL: https://github.com/apache/iceberg/pull/10020#discussion_r1593058306 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -140,6 +141,31 @@ public StructType partitionType() { return lazyPartitionType; } +

Re: [I] Add support for Categoricals as a data type in the writer [iceberg-python]

2024-05-07 Thread via GitHub
Fokko closed issue #450: Add support for Categoricals as a data type in the writer URL: https://github.com/apache/iceberg-python/issues/450 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Add support for categorical type [iceberg-python]

2024-05-07 Thread via GitHub
Fokko merged PR #693: URL: https://github.com/apache/iceberg-python/pull/693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Support `register_table` in GlueCatalog [iceberg-python]

2024-05-07 Thread via GitHub
MehulBatra commented on issue #705: URL: https://github.com/apache/iceberg-python/issues/705#issuecomment-2099154077 Initial Progress: https://github.com/apache/iceberg-python/pull/711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [FEAT]register table using iceberg metadata file via pyiceberg [iceberg-python]

2024-05-07 Thread via GitHub
MehulBatra opened a new pull request, #711: URL: https://github.com/apache/iceberg-python/pull/711 - If a user has an existing Iceberg table that is not yet registered in the AWS Glue Catalog, and they want to register that table in the Glue Catalog for better future access and management.

Re: [PR] Add ManifestFile Stats in snapshot summary. [iceberg]

2024-05-07 Thread via GitHub
nk1506 commented on code in PR #10246: URL: https://github.com/apache/iceberg/pull/10246#discussion_r1592861026 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -156,6 +156,8 @@ public List apply(TableMetadata base, Snapshot snapshot) { manifests.addAll(sn

Re: [PR] Support Appends with TimeTransform Partitions [iceberg-python]

2024-05-07 Thread via GitHub
syun64 commented on code in PR #703: URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1592831066 ## pyiceberg/transforms.py: ## @@ -349,6 +356,13 @@ def dedup_name(self) -> str: def preserves_order(self) -> bool: return True +@abstractmethod

Re: [PR] Build: Bump ray from 2.9.2 to 2.20.0 [iceberg-python]

2024-05-07 Thread via GitHub
HonahX commented on PR #691: URL: https://github.com/apache/iceberg-python/pull/691#issuecomment-2098908770 @dependabot ignore this minor version -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
stevenzwu commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1592780335 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to

Re: [PR] Build: Bump ray from 2.9.2 to 2.20.0 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] commented on PR #691: URL: https://github.com/apache/iceberg-python/pull/691#issuecomment-2098908873 OK, I won't notify you about version 2.20.x again, unless you re-open this PR. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Build: Bump ray from 2.9.2 to 2.20.0 [iceberg-python]

2024-05-07 Thread via GitHub
dependabot[bot] closed pull request #691: Build: Bump ray from 2.9.2 to 2.20.0 URL: https://github.com/apache/iceberg-python/pull/691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Support Appends with TimeTransform Partitions [iceberg-python]

2024-05-07 Thread via GitHub
HonahX commented on code in PR #703: URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1592801119 ## pyiceberg/transforms.py: ## @@ -349,6 +356,13 @@ def dedup_name(self) -> str: def preserves_order(self) -> bool: return True +@abstractmethod

Re: [PR] Support Appends with TimeTransform Partitions [iceberg-python]

2024-05-07 Thread via GitHub
HonahX commented on code in PR #703: URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1592801119 ## pyiceberg/transforms.py: ## @@ -349,6 +356,13 @@ def dedup_name(self) -> str: def preserves_order(self) -> bool: return True +@abstractmethod

Re: [PR] Make `add_files` to support `snapshot_properties` argument [iceberg-python]

2024-05-07 Thread via GitHub
HonahX merged PR #695: URL: https://github.com/apache/iceberg-python/pull/695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Make `add_files` to support `snapshot_properties` argument [iceberg-python]

2024-05-07 Thread via GitHub
HonahX commented on PR #695: URL: https://github.com/apache/iceberg-python/pull/695#issuecomment-2098885056 @enkidulan Thanks for fixing the test and the great work! Thanks @syun64 and @kevinjqliu for reviewing. Merging! -- This is an automated message from the Apache Git Service. To resp

Re: [PR] feat: Extract FileRead and FileWrite trait [iceberg-rust]

2024-05-07 Thread via GitHub
Xuanwo commented on PR #364: URL: https://github.com/apache/iceberg-rust/pull/364#issuecomment-2098851128 cc @Fokko @liurenjie1024 @ZENOTME for reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core: Introduce AuthConfig [iceberg]

2024-05-07 Thread via GitHub
nastra commented on code in PR #10161: URL: https://github.com/apache/iceberg/pull/10161#discussion_r1592491274 ## aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java: ## @@ -213,12 +214,13 @@ private AuthSession authSession() { ex

Re: [PR] mr: Fix using Date type as partition field [iceberg]

2024-05-07 Thread via GitHub
nastra merged PR #10210: URL: https://github.com/apache/iceberg/pull/10210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] docs: Add links checker [iceberg]

2024-05-07 Thread via GitHub
nastra commented on PR #9965: URL: https://github.com/apache/iceberg/pull/9965#issuecomment-2098413421 @bitsondatadev could you take a look at the issue that @print-sid8 reported please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[I] Cannot access table endpoint in REST catalog when table name contains a slash character (`/`) [iceberg-python]

2024-05-07 Thread via GitHub
RoseGoldIsntGay opened a new issue, #710: URL: https://github.com/apache/iceberg-python/issues/710 ### Apache Iceberg version 0.4.0 ### Please describe the bug ๐Ÿž When trying to get a table containing a slash character in the name from the Iceberg REST catalog, pyiceberg

Re: [PR] Support Appends with TimeTransform Partitions [iceberg-python]

2024-05-07 Thread via GitHub
syun64 commented on code in PR #703: URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1592449680 ## pyiceberg/transforms.py: ## @@ -349,6 +356,13 @@ def dedup_name(self) -> str: def preserves_order(self) -> bool: return True +@abstractmethod

Re: [PR] Support Appends with TimeTransform Partitions [iceberg-python]

2024-05-07 Thread via GitHub
syun64 commented on code in PR #703: URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1592430523 ## pyiceberg/transforms.py: ## @@ -349,6 +356,13 @@ def dedup_name(self) -> str: def preserves_order(self) -> bool: return True +@abstractmethod

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1592277579 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-05-07 Thread via GitHub
ajantha-bhat commented on PR #10176: URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2098057287 ping @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] docs: Add links checker [iceberg]

2024-05-07 Thread via GitHub
print-sid8 commented on PR #9965: URL: https://github.com/apache/iceberg/pull/9965#issuecomment-2097911026 Hi guys, Just reporting something I have noticed in the docs website of Iceberg for versions 1.5.0 and 1.5.1 and also the 'latest' section Lets say I go to - https://i

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-07 Thread via GitHub
wgtmac commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1592131896 ## format/materialized-view-spec.md: ## @@ -0,0 +1,55 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool to ab

Re: [PR] Flink: Read parquet BINARY column as String for expected [iceberg]

2024-05-07 Thread via GitHub
chenwyi2 commented on PR #8808: URL: https://github.com/apache/iceberg/pull/8808#issuecomment-2097906401 > > > * It seems that ORC is not experiencing this issue because it creates value reader based on the iceberg column types. > > > * Avro reads the fields entirely based on the file typ

Re: [PR] Test out Parquet 1.14.0 [iceberg]

2024-05-07 Thread via GitHub
jbonofre commented on PR #10209: URL: https://github.com/apache/iceberg/pull/10209#issuecomment-2097834845 Yes, I will move forward on the Gradle/revapi PR (a bit busy with a few Avro stuff ๐Ÿ˜„ ). -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Hive catalog: Add retry logic for hive locking [iceberg-python]

2024-05-07 Thread via GitHub
frankliee commented on code in PR #701: URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1592047115 ## pyiceberg/catalog/hive.py: ## @@ -356,11 +371,46 @@ def _create_lock_request(self, database_name: str, table_name: str) -> LockReque return lock_r

Re: [PR] Hive catalog: Add retry logic for hive locking [iceberg-python]

2024-05-07 Thread via GitHub
frankliee commented on code in PR #701: URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1592047115 ## pyiceberg/catalog/hive.py: ## @@ -356,11 +371,46 @@ def _create_lock_request(self, database_name: str, table_name: str) -> LockReque return lock_r

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592011406 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592009893 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592010621 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkCommitter.java: ## @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592008470 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergFlinkManifest.java: ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software F

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592008844 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergFlinkManifest.java: ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Data: Adopt EncryptingFileIO in Java read API [iceberg]

2024-05-07 Thread via GitHub
hsiang-c closed pull request #10201: Data: Adopt EncryptingFileIO in Java read API URL: https://github.com/apache/iceberg/pull/10201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Data: Adopt EncryptingFileIO in Java read API [iceberg]

2024-05-07 Thread via GitHub
hsiang-c commented on PR #10201: URL: https://github.com/apache/iceberg/pull/10201#issuecomment-2097719895 Spoke to Gidon offline and this PR is redundant in the latest design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592006641 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -0,0 +1,399 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592005960 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -0,0 +1,399 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1592004404 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/writer/IcebergStreamWriterMetrics.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591998705 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/SinkCommitter.java: ## @@ -0,0 +1,478 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591995708 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/SinkAggregator.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-05-07 Thread via GitHub
pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591993745 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergManifestOutputFileFactory.java: ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Hive catalog: Add retry logic for hive locking [iceberg-python]

2024-05-07 Thread via GitHub
frankliee commented on code in PR #701: URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591985541 ## tests/integration/test_reads.py: ## @@ -506,3 +508,40 @@ def test_hive_locking(session_catalog_hive: HiveCatalog) -> None: table.transactio

  1   2   >