Re: [I] gc.enabled property is set to false by default for Apache Iceberg table created in Nessie Catalog [iceberg]

2024-01-28 Thread via GitHub
nastra commented on issue #9562: URL: https://github.com/apache/iceberg/issues/9562#issuecomment-1914115059 @Ashwin07 this is in purpose as Nessie doesn't support Iceberg's GC operations due to the way Nessie manages metadata. You'd rather want to use https://projectnessie.org/features/mana

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-28 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1469148875 ## crates/iceberg/src/io.rs: ## @@ -278,7 +288,7 @@ impl OutputFile { } /// Creates output file for writing. -pub async fn writer(&self) -> Result {

[PR] feat: add parquet writer [iceberg-rust]

2024-01-28 Thread via GitHub
ZENOTME opened a new pull request, #176: URL: https://github.com/apache/iceberg-rust/pull/176 This PR adds a parquet writer under file writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build/Release: Upgrade to RAT 0.16.1 [iceberg]

2024-01-28 Thread via GitHub
jbonofre commented on PR #9579: URL: https://github.com/apache/iceberg/pull/9579#issuecomment-1914053599 @nastra @Fokko If you have some time to take a look, it would be great :) I did RAT 0.16.1 release with important performance boost in our case. -- This is an automated message from th

Re: [PR] [DO NOT MERGE] New docs switch [iceberg]

2024-01-28 Thread via GitHub
bitsondatadev commented on PR #9520: URL: https://github.com/apache/iceberg/pull/9520#issuecomment-1914011168 @rdblue I split out the directory changes: https://github.com/apache/iceberg/pull/9578 Now this PR merge just changes the site back. -- This is an automated message from th

[PR] Move nightly versioned docs to top-level docs directory [iceberg]

2024-01-28 Thread via GitHub
bitsondatadev opened a new pull request, #9578: URL: https://github.com/apache/iceberg/pull/9578 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1469080603 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -523,6 +523,78 @@ public void testMergeWithOnlyUpdateCla

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-28 Thread via GitHub
emkornfield commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1469066912 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-28 Thread via GitHub
advancedxy commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1469025897 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`|

Re: [PR] Spark 3.5: Fix testDeleteFileThenMetadataDelete failure due to table not refreshed [iceberg]

2024-01-28 Thread via GitHub
manuzhang commented on code in PR #9551: URL: https://github.com/apache/iceberg/pull/9551#discussion_r1468992361 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -166,6 +166,28 @@ public static Object[][

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468971880 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +986,76 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIce

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468986213 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +986,76 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIce

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468971880 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +986,76 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIce

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468971880 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +986,76 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIce

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468964393 ## tests/catalog/test_base.py: ## @@ -330,6 +333,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468964637 ## tests/catalog/test_base.py: ## @@ -330,6 +333,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468964637 ## tests/catalog/test_base.py: ## @@ -330,6 +333,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] Check the types when writing [iceberg-python]

2024-01-28 Thread via GitHub
HonahX merged PR #313: URL: https://github.com/apache/iceberg-python/pull/313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Small getting started guide on writes [iceberg-python]

2024-01-28 Thread via GitHub
HonahX commented on code in PR #311: URL: https://github.com/apache/iceberg-python/pull/311#discussion_r1468958598 ## mkdocs/docs/index.md: ## @@ -38,36 +38,129 @@ You can install the latest release version from pypi: pip install "pyiceberg[s3fs,hive]" ``` -Install it direct

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on PR #9176: URL: https://github.com/apache/iceberg/pull/9176#issuecomment-1913738824 @amogh-jahagirdar, I fixed the implementation in the PR above. It would be great to get this into 1.5.0 also! -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9176: URL: https://github.com/apache/iceberg/pull/9176#discussion_r1468947657 ## api/src/main/java/org/apache/iceberg/expressions/ValueAggregate.java: ## @@ -30,13 +30,16 @@ protected ValueAggregate(Operation op, BoundTerm term) { @Override

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-28 Thread via GitHub
jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1913734869 @rdblue it's reasonable. Let me update the PR to add an option to migrate database schema and enable view. Thanks ! -- This is an automated message from the Apache Git Service. To resp

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9176: URL: https://github.com/apache/iceberg/pull/9176#discussion_r1468943381 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java: ## @@ -249,6 +250,78 @@ public void testAggregateNotPushDownIfOneCantPushDown(

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9176: URL: https://github.com/apache/iceberg/pull/9176#discussion_r1468943381 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java: ## @@ -249,6 +250,78 @@ public void testAggregateNotPushDownIfOneCantPushDown(

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1468941095 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -60,17 +65,27 @@ case class RewriteViewCommands(spa

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1468939586 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateV2ViewExec.scala: ## @@ -0,0 +1,79 @@ +/* + * Licensed to the Apach

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1468939586 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateV2ViewExec.scala: ## @@ -0,0 +1,79 @@ +/* + * Licensed to the Apach

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1468939456 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala: ## @@ -123,6 +132,47 @@ case class Extend

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468937534 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala: ## @@ -123,6 +127,35 @@ case class Extend

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468937382 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/views/ResolvedV2View.scala: ## @@ -22,10 +22,12 @@ package org.apache.spark.sql

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468936985 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala: ## @@ -123,6 +127,35 @@ case class Extend

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468936762 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -60,17 +63,23 @@ case class RewriteViewCommands(spa

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468935466 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -123,4 +132,17 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468935203 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -123,4 +132,17 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468933314 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckViews.scala: ## @@ -36,6 +38,9 @@ object CheckViews extends (LogicalPlan => Uni

Re: [PR] Spark: Support altering views [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1468933314 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckViews.scala: ## @@ -36,6 +38,9 @@ object CheckViews extends (LogicalPlan => Uni

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1913713816 @jbonofre, I think @danielcweeks and I are both on the side of needing to have a single table. The less bad option is to have older clients fail to read metadata. There is no other solutio

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1468932165 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -214,6 +214,8 @@ object RewriteMergeIntoTable ext

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1468928409 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -214,6 +214,8 @@ object RewriteMergeIntoTable ext

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1468925515 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -214,6 +214,8 @@ object RewriteMergeIntoTable ext

Re: [PR] [DO NOT MERGE] New docs switch [iceberg]

2024-01-28 Thread via GitHub
bitsondatadev commented on PR #9520: URL: https://github.com/apache/iceberg/pull/9520#issuecomment-1913687204 > @bitsondatadev, the changes to move over to the site under `site` and the changes to `docs` must be separate PRs. We have not yet fixed the nightly docs, so I think we should remo

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-28 Thread via GitHub
emkornfield commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1468916964 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1468915010 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`|`"mo

Re: [PR] [DO NOT MERGE] New docs switch [iceberg]

2024-01-28 Thread via GitHub
rdblue commented on PR #9520: URL: https://github.com/apache/iceberg/pull/9520#issuecomment-1913675812 @bitsondatadev, the changes to move over to the site under `site` and the changes to `docs` must be separate PRs. We have not yet fixed the nightly docs, so I think we should remove them.

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-01-28 Thread via GitHub
javrasya commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-1913636569 Do you want me to proceed and update my PR @pvary accordingly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[PR] chore(deps): Update derive_builder requirement from 0.12.0 to 0.13.0 [iceberg-rust]

2024-01-28 Thread via GitHub
dependabot[bot] opened a new pull request, #175: URL: https://github.com/apache/iceberg-rust/pull/175 Updates the requirements on [derive_builder](https://github.com/colin-kiegel/rust-derive-builder) to permit the latest version. Release notes Sourced from https://github.com/colin

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.26.3 to 1.26.5 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] closed pull request #52: build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.26.3 to 1.26.5 URL: https://github.com/apache/iceberg-go/pull/52 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.26.3 to 1.26.5 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] commented on PR #52: URL: https://github.com/apache/iceberg-go/pull/52#issuecomment-1913515156 Superseded by #57. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.26.3 to 1.26.6 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] opened a new pull request, #57: URL: https://github.com/apache/iceberg-go/pull/57 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.26.3 to 1.26.6. Commits https://github.com/aws/aws-sdk-go-v2/commit/857d5277b41f290ab7ba0e

Re: [PR] build(deps): bump github.com/hamba/avro/v2 from 2.16.0 to 2.18.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] commented on PR #37: URL: https://github.com/apache/iceberg-go/pull/37#issuecomment-1913514938 Superseded by #56. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[PR] build(deps): bump github.com/hamba/avro/v2 from 2.16.0 to 2.19.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] opened a new pull request, #56: URL: https://github.com/apache/iceberg-go/pull/56 Bumps [github.com/hamba/avro/v2](https://github.com/hamba/avro) from 2.16.0 to 2.19.0. Release notes Sourced from https://github.com/hamba/avro/releases";>github.com/hamba/avro/v2's r

[PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.6.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] opened a new pull request, #55: URL: https://github.com/apache/iceberg-go/pull/55 Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.3.1 to 1.6.0. Release notes Sourced from https://github.com/google/uuid/releases";>github.com/google/uuid's relea

Re: [PR] build(deps): bump github.com/hamba/avro/v2 from 2.16.0 to 2.18.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] closed pull request #37: build(deps): bump github.com/hamba/avro/v2 from 2.16.0 to 2.18.0 URL: https://github.com/apache/iceberg-go/pull/37 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.5.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] commented on PR #46: URL: https://github.com/apache/iceberg-go/pull/46#issuecomment-1913514902 Superseded by #55. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.5.0 [iceberg-go]

2024-01-28 Thread via GitHub
dependabot[bot] closed pull request #46: build(deps): bump github.com/google/uuid from 1.3.1 to 1.5.0 URL: https://github.com/apache/iceberg-go/pull/46 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t