Re: [PR] Build: Bump mkdocs-material from 9.6.7 to 9.6.8 [iceberg]

2025-03-15 Thread via GitHub
Fokko merged PR #12542: URL: https://github.com/apache/iceberg/pull/12542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump datamodel-code-generator from 0.28.2 to 0.28.4 [iceberg]

2025-03-15 Thread via GitHub
Fokko merged PR #12541: URL: https://github.com/apache/iceberg/pull/12541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Unable to read some Iceberg V1 tables [iceberg-rust]

2025-03-15 Thread via GitHub
phillipleblanc commented on issue #1088: URL: https://github.com/apache/iceberg-rust/issues/1088#issuecomment-2727198817 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] Unable to read some Iceberg V1 tables [iceberg-rust]

2025-03-15 Thread via GitHub
phillipleblanc opened a new issue, #1088: URL: https://github.com/apache/iceberg-rust/issues/1088 ### Apache Iceberg Rust version None ### Describe the bug There are some valid Iceberg V1 tables that cannot be read using `iceberg-rust` ### To Reproduce This

[PR] Make `schema` and `partition_spec` optional for TableMetadataV1 [iceberg-rust]

2025-03-15 Thread via GitHub
phillipleblanc opened a new pull request, #1087: URL: https://github.com/apache/iceberg-rust/pull/1087 ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Fixes an issue reading V1 Iceberg tables created by AWS Glue. The table

[PR] Build: Bump mkdocs-material from 9.6.7 to 9.6.8 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12542: URL: https://github.com/apache/iceberg/pull/12542 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.7 to 9.6.8. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-

[PR] Build: Bump datamodel-code-generator from 0.28.2 to 0.28.4 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12541: URL: https://github.com/apache/iceberg/pull/12541 Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.28.2 to 0.28.4. Release notes Sourced from https://github.com/koxudaxi/datamodel-code-

[PR] Core: fix "Unclosed S3FileIO" warning in JdbcCatalog [iceberg]

2025-03-15 Thread via GitHub
rcjverhoef opened a new pull request, #12540: URL: https://github.com/apache/iceberg/pull/12540 See https://github.com/apache/iceberg/issues/12539 I am adding the `io` to the `closeableGroup` with this PR. Any FileIO that is opened should ideally be closed and not left to linger for t

Re: [PR] Build: Bump junit-platform from 1.11.4 to 1.12.0 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] closed pull request #12381: Build: Bump junit-platform from 1.11.4 to 1.12.0 URL: https://github.com/apache/iceberg/pull/12381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] Build: Bump software.amazon.awssdk:bom from 2.30.31 to 2.31.1 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12536: URL: https://github.com/apache/iceberg/pull/12536 Bumps software.amazon.awssdk:bom from 2.30.31 to 2.31.1. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softw

Re: [PR] Build: Bump junit-platform from 1.11.4 to 1.12.0 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] commented on PR #12381: URL: https://github.com/apache/iceberg/pull/12381#issuecomment-2727176652 Superseded by #12538. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] Build: Bump junit-platform from 1.11.4 to 1.12.1 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12538: URL: https://github.com/apache/iceberg/pull/12538 Bumps `junit-platform` from 1.11.4 to 1.12.1. Updates `org.junit.platform:junit-platform-suite-api` from 1.11.4 to 1.12.1 Commits See full diff in https://github.com/junit-tea

[PR] Build: Bump junit from 5.11.4 to 5.12.1 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12537: URL: https://github.com/apache/iceberg/pull/12537 Bumps `junit` from 5.11.4 to 5.12.1. Updates `org.junit.jupiter:junit-jupiter` from 5.11.4 to 5.12.1 Release notes Sourced from https://github.com/junit-team/junit5/releases";>org

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.1 [iceberg]

2025-03-15 Thread via GitHub
dependabot[bot] opened a new pull request, #12535: URL: https://github.com/apache/iceberg/pull/12535 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.23.0 to 3.23.1. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/

Re: [PR] Spark: DVs + Positional Deletes + Compaction [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on PR #12181: URL: https://github.com/apache/iceberg/pull/12181#issuecomment-2727079281 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on PR #12240: URL: https://github.com/apache/iceberg/pull/12240#issuecomment-2727079301 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatโ€™s incorrect or this pul

Re: [PR] Spark: DVs + Positional Deletes + Compaction [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] closed pull request #12181: Spark: DVs + Positional Deletes + Compaction URL: https://github.com/apache/iceberg/pull/12181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Open-API: Fix compilation errors in generated Java classes due to mismatched return types [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] closed pull request #11806: Open-API: Fix compilation errors in generated Java classes due to mismatched return types URL: https://github.com/apache/iceberg/pull/11806 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Open-API: Fix compilation errors in generated Java classes due to mismatched return types [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on PR #11806: URL: https://github.com/apache/iceberg/pull/11806#issuecomment-2727079260 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2025-03-15 Thread via GitHub
github-actions[bot] commented on issue #11016: URL: https://github.com/apache/iceberg/issues/11016#issuecomment-2727079215 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2025-03-15 Thread via GitHub
github-actions[bot] closed issue #11016: Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID = -3757309436761247407 URL: https://github.com/apache/iceberg

Re: [I] How to set Spark conf to use Parquet and Iceberg tables using glue catalog without catalog name(spark_catalog)? [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on issue #7748: URL: https://github.com/apache/iceberg/issues/7748#issuecomment-2727079178 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to set Spark conf to use Parquet and Iceberg tables using glue catalog without catalog name(spark_catalog)? [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] closed issue #7748: How to set Spark conf to use Parquet and Iceberg tables using glue catalog without catalog name(spark_catalog)? URL: https://github.com/apache/iceberg/issues/7748 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] AWS: provide option to hide old fields in Glue table [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] closed issue #7584: AWS: provide option to hide old fields in Glue table URL: https://github.com/apache/iceberg/issues/7584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] AWS: provide option to hide old fields in Glue table [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on issue #7584: URL: https://github.com/apache/iceberg/issues/7584#issuecomment-2727079168 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[PR] feat: (catalog/glue): Fix glue table type [iceberg-go]

2025-03-15 Thread via GitHub
dttung2905 opened a new pull request, #333: URL: https://github.com/apache/iceberg-go/pull/333 Hi team, I made a dumb mistake in https://github.com/apache/iceberg-go/pull/326 that the table created is a standard AWS Glue Table.It should ,however, be Apache Iceberg-type table instead. I'm

Re: [PR] Update-schema: Add support for `initial-default` [iceberg-python]

2025-03-15 Thread via GitHub
sungwy commented on PR #1770: URL: https://github.com/apache/iceberg-python/pull/1770#issuecomment-2726991644 @Fokko the PR looks good to me: I think we may just have missed including the new properties in the `rename_column` method. I agree that we could introduce the ability to update `wr

Re: [PR] Update-schema: Add support for `initial-default` [iceberg-python]

2025-03-15 Thread via GitHub
sungwy commented on code in PR #1770: URL: https://github.com/apache/iceberg-python/pull/1770#discussion_r1997301238 ## pyiceberg/table/update/schema.py: ## @@ -212,13 +215,34 @@ def add_column( # assign new IDs in order new_id = self.assign_new_column_id() +

Re: [PR] WIP: Implement readers and writers for new types [iceberg]

2025-03-15 Thread via GitHub
rdblue closed pull request #12407: WIP: Implement readers and writers for new types URL: https://github.com/apache/iceberg/pull/12407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] WIP: Implement readers and writers for new types [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on PR #12407: URL: https://github.com/apache/iceberg/pull/12407#issuecomment-2726985151 Closing this. The changes were broken across #12532, #12470, #12463, #12455, and #12323. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Flink 1.18, 1.19: Implement timestamp(9), unknown, and defaults [iceberg]

2025-03-15 Thread via GitHub
rdblue merged PR #12532: URL: https://github.com/apache/iceberg/pull/12532 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Flink 1.18, 1.19: Implement timestamp(9), unknown, and defaults [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on PR #12532: URL: https://github.com/apache/iceberg/pull/12532#issuecomment-2726984418 Thanks, @danielcweeks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] Support transforms with datetime timezones [iceberg-rust]

2025-03-15 Thread via GitHub
kevinjqliu opened a new pull request, #1086: URL: https://github.com/apache/iceberg-rust/pull/1086 ## Which issue does this PR close? - Closes #1078. ## What changes are included in this PR? Enable `chrono-tz` feature in the `arrow` dependency ## Are th

Re: [PR] V3: Introduce `timestamp_ns` and `timestamptz_ns` [iceberg-python]

2025-03-15 Thread via GitHub
sungwy commented on code in PR #1632: URL: https://github.com/apache/iceberg-python/pull/1632#discussion_r1997260223 ## pyiceberg/utils/datetime.py: ## @@ -91,6 +91,57 @@ def timestamp_to_micros(timestamp_str: str) -> int: raise ValueError(f"Invalid timestamp without zone:

Re: [PR] V3: Introduce `timestamp_ns` and `timestamptz_ns` [iceberg-python]

2025-03-15 Thread via GitHub
sungwy commented on code in PR #1632: URL: https://github.com/apache/iceberg-python/pull/1632#discussion_r1997259581 ## pyiceberg/types.py: ## @@ -175,6 +179,10 @@ def is_primitive(self) -> bool: def is_struct(self) -> bool: return isinstance(self, StructType) +

Re: [PR] Add pull-request template [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1777: URL: https://github.com/apache/iceberg-python/pull/1777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Kafka Connect: Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-15 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1995774477 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/MockIcebergSinkTask.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1996309678 ## core/src/test/java/org/apache/iceberg/data/DataTest.java: ## @@ -129,28 +135,37 @@ protected boolean supportsVariant() { return false; } + protected bool

Re: [PR] Core: Use `buildKeepingLast` for table properties in REST table builder [iceberg]

2025-03-15 Thread via GitHub
nastra merged PR #12526: URL: https://github.com/apache/iceberg/pull/12526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat: (catalog/glue) Add support for CreateTable [iceberg-go]

2025-03-15 Thread via GitHub
zeroshade commented on code in PR #326: URL: https://github.com/apache/iceberg-go/pull/326#discussion_r1996108014 ## catalog/glue/schema.go: ## @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the N

Re: [PR] Bump version to 0.10.0 [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1791: URL: https://github.com/apache/iceberg-python/pull/1791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Build: Bump mkdocs-material from 9.6.7 to 9.6.8 [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1792: URL: https://github.com/apache/iceberg-python/pull/1792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Kafka Connect: Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-15 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1993608421 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java: ## @@ -43,6 +43,21 @@ public class CommitterImpl implements Commit

Re: [PR] Build: Bump mypy-boto3-glue from 1.37.0 to 1.37.13 [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1795: URL: https://github.com/apache/iceberg-python/pull/1795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Core: Bulk deletion in RemoveSnapshots [iceberg]

2025-03-15 Thread via GitHub
gaborkaszab commented on code in PR #11837: URL: https://github.com/apache/iceberg/pull/11837#discussion_r1993832511 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1621,6 +1627,82 @@ public void testRetainFilesOnRetainedBranches() { assertThat(del

Re: [PR] Build: Bump polars from 1.24.0 to 1.25.0 [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1794: URL: https://github.com/apache/iceberg-python/pull/1794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996040055 ## core/src/main/java/org/apache/iceberg/io/datafile/ReadBuilder.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Build: Bump duckdb from 1.2.0 to 1.2.1 [iceberg-python]

2025-03-15 Thread via GitHub
kevinjqliu merged PR #1793: URL: https://github.com/apache/iceberg-python/pull/1793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r1996413461 ## core/src/main/java/org/apache/iceberg/variants/ShreddedArray.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996026806 ## core/src/main/java/org/apache/iceberg/io/datafile/DataFileServiceRegistry.java: ## @@ -0,0 +1,231 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[I] Refactor setting the `max_changed_partitions_for_summaries` [iceberg-python]

2025-03-15 Thread via GitHub
Fokko opened a new issue, #1779: URL: https://github.com/apache/iceberg-python/issues/1779 ### Feature Request / Improvement I think it would be better to set the partition-limit in the `__init__` of `SnapshotSummaryCollector`: https://github.com/apache/iceberg-python/blob/764

Re: [I] Multi-Column Transforms [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on issue #9132: URL: https://github.com/apache/iceberg/issues/9132#issuecomment-2712135065 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r1996396090 ## core/src/test/java/org/apache/iceberg/variants/TestShreddedArray.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Auth Manager API part 6: API enablement [iceberg]

2025-03-15 Thread via GitHub
nastra commented on code in PR #12197: URL: https://github.com/apache/iceberg/pull/12197#discussion_r1995609891 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1206,12 +1006,15 @@ public void commitTransaction(SessionContext context, List commits)

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-03-15 Thread via GitHub
Xuanwo merged PR #965: URL: https://github.com/apache/iceberg-rust/pull/965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Support `wasb://` and `wasbs://` [iceberg-python]

2025-03-15 Thread via GitHub
christophediprima commented on PR #1663: URL: https://github.com/apache/iceberg-python/pull/1663#issuecomment-2710950370 We have been testing it with my team and we had no issues. What kind of tests can you think about? -- This is an automated message from the Apache Git Service. To respo

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996047949 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseBatchReader.java: ## @@ -65,76 +59,32 @@ protected CloseableIterable newBatchIterable( Exp

Re: [PR] feat: (catalog/glue) Add support for CreateTable [iceberg-go]

2025-03-15 Thread via GitHub
zeroshade commented on code in PR #326: URL: https://github.com/apache/iceberg-go/pull/326#discussion_r1987502468 ## catalog/glue/glue_test.go: ## @@ -778,12 +781,103 @@ func TestGlueListNamespacesIntegration(t *testing.T) { } assert := require.New(t) - a

Re: [PR] Spark 3.5: Add Comet tests [iceberg]

2025-03-15 Thread via GitHub
github-actions[bot] commented on PR #12176: URL: https://github.com/apache/iceberg/pull/12176#issuecomment-2726047328 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Getting the following error when using from spark thrift server [iceberg]

2025-03-15 Thread via GitHub
dvaldivia commented on issue #3010: URL: https://github.com/apache/iceberg/issues/3010#issuecomment-2705028195 I suspect the problem is that `start-thriftserver.sh` doesn't load the dynamic dependencies via `--packages` the way `spark-shell` does -- This is an automated message from the A

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996050564 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseBatchReader.java: ## @@ -65,76 +59,32 @@ protected CloseableIterable newBatchIterable( Exp

[PR] Core: Fix support for GenericManifestFile intex projection [iceberg]

2025-03-15 Thread via GitHub
danielcweeks opened a new pull request, #12522: URL: https://github.com/apache/iceberg/pull/12522 This PR adds the proper constructor to allow use of the `SupportsIndexProjection` with `GenericManifestFile`. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996055858 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DataFileServices.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] AWS: Update S3 async client configurations and docs for analytics-accelerator-s3 [iceberg]

2025-03-15 Thread via GitHub
SanjayMarreddi commented on code in PR #12503: URL: https://github.com/apache/iceberg/pull/12503#discussion_r1991923527 ## docs/docs/aws.md: ## @@ -565,6 +565,29 @@ spark-sql --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCata For more details on using S3

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
Kontinuation commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1995662918 ## api/src/main/java/org/apache/iceberg/types/EdgeInterpolationAlgorithm.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
Kontinuation commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1994563992 ## core/src/main/java/org/apache/iceberg/SchemaParser.java: ## @@ -141,11 +145,39 @@ static void toJson(Types.MapType map, JsonGenerator generator) throws IOExc

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
szehon-ho commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1995866592 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +563,122 @@ public int hashCode() { } } + public static class GeometryType extends

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
Kontinuation commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1995663888 ## core/src/test/java/org/apache/iceberg/TestGeospatialTable.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Docs: Update Iceberg talks with recent Iceberg meetup sessions [iceberg]

2025-03-15 Thread via GitHub
nastra merged PR #12481: URL: https://github.com/apache/iceberg/pull/12481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996102983 ## core/src/main/java/org/apache/iceberg/io/datafile/WriteBuilder.java: ## @@ -0,0 +1,340 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Auth Manager API part 6: API enablement [iceberg]

2025-03-15 Thread via GitHub
nastra commented on code in PR #12197: URL: https://github.com/apache/iceberg/pull/12197#discussion_r1995612671 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1512,10 +1340,16 @@ private View replace(LoadViewResponse response) { ViewMetada

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996110305 ## core/src/main/java/org/apache/iceberg/io/datafile/WriteBuilder.java: ## @@ -0,0 +1,340 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Fix default and initial value handling on table creation [iceberg]

2025-03-15 Thread via GitHub
nastra commented on code in PR #12520: URL: https://github.com/apache/iceberg/pull/12520#discussion_r1995246392 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -658,6 +659,38 @@ public void testCompleteCreateTable() { .isEqualTo(UUID.fromString

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1994443479 ## build.gradle: ## @@ -348,6 +348,7 @@ project(':iceberg-core') { implementation libs.jackson.databind implementation libs.caffeine implementation libs

Re: [I] Applying Filter on Top-Level Struct Columns Throws Error [iceberg-python]

2025-03-15 Thread via GitHub
Fokko commented on issue #1778: URL: https://github.com/apache/iceberg-python/issues/1778#issuecomment-2711728107 @srilman Thanks for raising this and the excellent issue. Looking at the stacktrace, this happens at the evaluation of the partitioning, but it seems like there is no partition

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1996296188 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -70,6 +76,25 @@ public static Type fromTypeName(String typeString) { return TYPES.get(lowerTypeS

Re: [I] How to understand "Partition evolution is a metadata operation and does not eagerly rewrite files." [iceberg]

2025-03-15 Thread via GitHub
madeirak commented on issue #12492: URL: https://github.com/apache/iceberg/issues/12492#issuecomment-2712381623 > It will not. New files will be written to the new spec, old files will remain in the old spec. > > Files from multiple partition specs can always exist in the same table.

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996027399 ## core/src/main/java/org/apache/iceberg/io/datafile/DataFileServiceRegistry.java: ## @@ -0,0 +1,231 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Core: Don't create empty RemovePartitionSpecs MetadataUpdate [iceberg]

2025-03-15 Thread via GitHub
nastra merged PR #12465: URL: https://github.com/apache/iceberg/pull/12465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Kafka Connect: Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-15 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1995821392 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java: ## @@ -46,26 +45,23 @@ public String version() { @Override public

Re: [PR] feat: (catalog/glue) Add support for CreateTable [iceberg-go]

2025-03-15 Thread via GitHub
zeroshade commented on code in PR #326: URL: https://github.com/apache/iceberg-go/pull/326#discussion_r1995759395 ## catalog/glue/glue_test.go: ## @@ -778,12 +781,103 @@ func TestGlueListNamespacesIntegration(t *testing.T) { } assert := require.New(t) - a

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1995900693 ## core/src/test/java/org/apache/iceberg/TestGeospatialTable.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [I] catalog table-default and table-override properties are not supported in CREATE_OR_REPLACE operation in IRC [iceberg]

2025-03-15 Thread via GitHub
puchengy commented on issue #12506: URL: https://github.com/apache/iceberg/issues/12506#issuecomment-2723719005 Hey @nastra, thanks for the suggestion. I think the fix won't solve my problem. In my case, the default table properties that I set is on **SERVER** side, and the change you share

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996063073 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DataFileServices.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996049790 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseBatchReader.java: ## @@ -65,76 +59,32 @@ protected CloseableIterable newBatchIterable( Exp

Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

2025-03-15 Thread via GitHub
deniskuzZ commented on code in PR #12528: URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329 ## api/src/main/java/org/apache/iceberg/types/Comparators.java: ## @@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) { return 0;

Re: [PR] feat: (catalog/glue) Add support for CreateTable [iceberg-go]

2025-03-15 Thread via GitHub
dttung2905 commented on code in PR #326: URL: https://github.com/apache/iceberg-go/pull/326#discussion_r1987616808 ## catalog/glue/glue.go: ## @@ -582,3 +633,16 @@ func filterDatabaseListByType(databases []types.Database, databaseType string) [ return filtered } + +f

Re: [I] Make reporter() function public from BaseTable public so that it can be used in Trino for creating BaseTable [iceberg]

2025-03-15 Thread via GitHub
pvary closed issue #12513: Make reporter() function public from BaseTable public so that it can be used in Trino for creating BaseTable URL: https://github.com/apache/iceberg/issues/12513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-03-15 Thread via GitHub
ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1993387221 ## core/src/main/java/org/apache/iceberg/SetPartitionStatistics.java: ## @@ -36,7 +35,10 @@ public SetPartitionStatistics(TableOperations ops) { @Override

[I] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

2025-03-15 Thread via GitHub
deniskuzZ opened a new issue, #12529: URL: https://github.com/apache/iceberg/issues/12529 ### Apache Iceberg version 1.8.1 (latest release) ### Query engine None ### Please describe the bug ๐Ÿž Below 3 StructProjections are considered equal leading to key over

Re: [I] Unable to set `write-default` for a column [iceberg]

2025-03-15 Thread via GitHub
malhotrashivam commented on issue #12495: URL: https://github.com/apache/iceberg/issues/12495#issuecomment-2711969219 Another issue with `write-default` is that there doesn't exist any APIs for updating the schema of an existing table to set the default value for any column, whereas the spe

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996038651 ## core/src/main/java/org/apache/iceberg/io/datafile/ReadBuilder.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Kafka Connect: Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-15 Thread via GitHub
bryanck commented on PR #12372: URL: https://github.com/apache/iceberg/pull/12372#issuecomment-2725277696 Thanks @kumarpritam863 for the research on this and the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] pass proxy configuration from environment vars to http client [iceberg]

2025-03-15 Thread via GitHub
akhilputhiry commented on code in PR #12406: URL: https://github.com/apache/iceberg/pull/12406#discussion_r1984518742 ## core/src/main/java/org/apache/iceberg/rest/RESTCatalog.java: ## @@ -55,7 +56,22 @@ public class RESTCatalog public RESTCatalog() { this( Sess

Re: [PR] Core: Set missing table-default property in RESTSessionCatalog [iceberg]

2025-03-15 Thread via GitHub
smaheshwar-pltr commented on PR #11646: URL: https://github.com/apache/iceberg/pull/11646#issuecomment-2724302288 Thanks for this PR! Looking forward to `table-override` being supported for REST catalogs too ๐Ÿ˜„ @ebyhr / @nastra - think I noticed a tiny bug (https://github.com/apache

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996104117 ## core/src/main/java/org/apache/iceberg/io/datafile/WriteBuilder.java: ## @@ -0,0 +1,340 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Fix default and initial value handling on table creation [iceberg]

2025-03-15 Thread via GitHub
pvary commented on code in PR #12520: URL: https://github.com/apache/iceberg/pull/12520#discussion_r1995311328 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java: ## @@ -84,7 +86,12 @@ protected void writeAndValidate(Schema writeSchema, Schema e

Re: [I] Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes [iceberg]

2025-03-15 Thread via GitHub
sfc-gh-ygu commented on issue #12467: URL: https://github.com/apache/iceberg/issues/12467#issuecomment-2706993884 It should not be allowed to apply pos deletes and eq deletes when `write.delete.mode = copy-on-write`. NO? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-03-15 Thread via GitHub
rdblue commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r1996058909 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DataFileServices.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] OpenAPI: Handle NamespaceNotEmptyException when dropping a namespace [iceberg]

2025-03-15 Thread via GitHub
nastra commented on code in PR #12518: URL: https://github.com/apache/iceberg/pull/12518#discussion_r1995883250 ## open-api/rest-catalog-open-api.yaml: ## @@ -4744,6 +4764,16 @@ components: } } +NamespaceNotEmptyError: + summary: The requested name

  1   2   >