Re: [PR] Fix error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-11-23 Thread via GitHub
a-agmon commented on PR #664: URL: https://github.com/apache/iceberg-rust/pull/664#issuecomment-2495833710 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Spec: add variant type [iceberg]

2024-11-23 Thread via GitHub
aihuaxu commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1855366984 ## format/spec.md: ## @@ -1154,6 +1169,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`struct`**|`record`|| |**`list`**|`

[PR] Build: Bump mkdocs-material from 9.5.44 to 9.5.45 [iceberg]

2024-11-23 Thread via GitHub
dependabot[bot] opened a new pull request, #11641: URL: https://github.com/apache/iceberg/pull/11641 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.44 to 9.5.45. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdoc

[PR] Build: Bump testcontainers from 1.20.3 to 1.20.4 [iceberg]

2024-11-23 Thread via GitHub
dependabot[bot] opened a new pull request, #11640: URL: https://github.com/apache/iceberg/pull/11640 Bumps `testcontainers` from 1.20.3 to 1.20.4. Updates `org.testcontainers:testcontainers` from 1.20.3 to 1.20.4 Release notes Sourced from https://github.com/testcontainers/testco

[PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.35.1 to 2.36.0 [iceberg]

2024-11-23 Thread via GitHub
dependabot[bot] opened a new pull request, #11638: URL: https://github.com/apache/iceberg/pull/11638 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.35.1 to 2.36.0. Release notes Sourced from https://github.com/google/error-prone

[PR] Build: Bump software.amazon.awssdk:bom from 2.29.15 to 2.29.20 [iceberg]

2024-11-23 Thread via GitHub
dependabot[bot] opened a new pull request, #11639: URL: https://github.com/apache/iceberg/pull/11639 Bumps software.amazon.awssdk:bom from 2.29.15 to 2.29.20. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

[PR] Build: Bump nessie from 0.100.0 to 0.100.2 [iceberg]

2024-11-23 Thread via GitHub
dependabot[bot] opened a new pull request, #11637: URL: https://github.com/apache/iceberg/pull/11637 Bumps `nessie` from 0.100.0 to 0.100.2. Updates `org.projectnessie.nessie:nessie-client` from 0.100.0 to 0.100.2 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-23 Thread via GitHub
RussellSpitzer commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855332823 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-23 Thread via GitHub
RussellSpitzer commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855332823 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||

Re: [PR] [SIP] fix error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-11-23 Thread via GitHub
FANNG1 commented on PR #664: URL: https://github.com/apache/iceberg-rust/pull/664#issuecomment-2495746322 @a-agmon @liurenjie1024 , sorry for the delay, it's ready to review now, please help to review when you have time, thanks -- This is an automated message from the Apache Git Service.

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #10953: URL: https://github.com/apache/iceberg/pull/10953#issuecomment-2495713845 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Add optional Glue Schema configuration to exclude Non-Current Fields [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #11334: URL: https://github.com/apache/iceberg/pull/11334#issuecomment-2495713873 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] API: Align CharSequenceSet impl with Data/DeleteFileSet [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed pull request #11322: API: Align CharSequenceSet impl with Data/DeleteFileSet URL: https://github.com/apache/iceberg/pull/11322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Core: Fix drop partition field and schema field error [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #11387: URL: https://github.com/apache/iceberg/pull/11387#issuecomment-2495713891 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Reimplement CharSequenceMap to obey Map contract [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed pull request #11308: Core: Reimplement CharSequenceMap to obey Map contract URL: https://github.com/apache/iceberg/pull/11308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Flink Support for TIMESTAMP_NANOS [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #11348: URL: https://github.com/apache/iceberg/pull/11348#issuecomment-2495713885 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Introduce a parameter to control whether the flink writer is linked with the previous operator [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on issue #10371: URL: https://github.com/apache/iceberg/issues/10371#issuecomment-2495713828 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Using the Iceberg catalog in your file system [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on issue #10326: URL: https://github.com/apache/iceberg/issues/10326#issuecomment-2495713800 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Quick notes how to update docs and javadoc at release publication time. [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #10810: URL: https://github.com/apache/iceberg/pull/10810#issuecomment-2495713835 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] API: Align CharSequenceSet impl with Data/DeleteFileSet [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #11322: URL: https://github.com/apache/iceberg/pull/11322#issuecomment-2495713864 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] catalog issue [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on issue #10324: URL: https://github.com/apache/iceberg/issues/10324#issuecomment-2495713791 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Core: Reimplement CharSequenceMap to obey Map contract [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #11308: URL: https://github.com/apache/iceberg/pull/11308#issuecomment-2495713858 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed pull request #10953: Arrow: add support for null vectors URL: https://github.com/apache/iceberg/pull/10953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Using subdirectory to dave data in ICEBERG. [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed issue #10327: Using subdirectory to dave data in ICEBERG. URL: https://github.com/apache/iceberg/issues/10327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Using subdirectory to dave data in ICEBERG. [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on issue #10327: URL: https://github.com/apache/iceberg/issues/10327#issuecomment-2495713811 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] catalog issue [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed issue #10324: catalog issue URL: https://github.com/apache/iceberg/issues/10324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Improvements to Iceberg Catalog Descriptions [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] closed issue #10316: Improvements to Iceberg Catalog Descriptions URL: https://github.com/apache/iceberg/issues/10316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Support different JDBC backend in the `JdbcCatalog` [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on issue #9733: URL: https://github.com/apache/iceberg/issues/9733#issuecomment-2495713743 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-11-23 Thread via GitHub
github-actions[bot] commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-2495713754 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] A more robust way to deprecate APIs [iceberg-python]

2024-11-23 Thread via GitHub
ndrluis commented on issue #1330: URL: https://github.com/apache/iceberg-python/issues/1330#issuecomment-2495711478 One new aspect introduced with this Conda deprecation code is the concept of a pending deprecation process. I think that when we release version 1.0, we will need to allocate

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-23 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1855301400 ## tests/table/test_init.py: ## @@ -982,28 +982,43 @@ def test_assert_table_uuid(table_v2: Table) -> None: def test_assert_ref_snapshot_id(table_v2: Table) -> No

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-23 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1855299774 ## pyiceberg/table/update/__init__.py: ## @@ -609,11 +609,14 @@ class AssertRefSnapshotId(ValidatableTableRequirement): type: Literal["assert-ref-snapshot-id

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-23 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1855299357 ## pyiceberg/table/__init__.py: ## @@ -1003,22 +1015,27 @@ def overwrite( overwrite_filter: ALWAYS_TRUE when you overwrite all the data,

Re: [PR] Introduce `assign_fresh_ids` flag and allow skipping fresh assignment of IDs on Table creation [iceberg-python]

2024-11-23 Thread via GitHub
sungwy commented on code in PR #1304: URL: https://github.com/apache/iceberg-python/pull/1304#discussion_r1855272561 ## pyiceberg/table/metadata.py: ## @@ -517,12 +517,15 @@ def new_table_metadata( location: str, properties: Properties = EMPTY_DICT, table_uuid: Op

Re: [I] Delete orphan files [iceberg-python]

2024-11-23 Thread via GitHub
kevinjqliu commented on issue #1200: URL: https://github.com/apache/iceberg-python/issues/1200#issuecomment-2495641285 That looks generally correct to me. There are a few caveats though. This assumes that the entire iceberg table (metadata and data files) is in a single location and that n

Re: [PR] Introduce `assign_fresh_ids` flag and allow skipping fresh assignment of IDs on Table creation [iceberg-python]

2024-11-23 Thread via GitHub
kevinjqliu commented on code in PR #1304: URL: https://github.com/apache/iceberg-python/pull/1304#discussion_r1855257218 ## pyiceberg/table/metadata.py: ## @@ -517,12 +517,15 @@ def new_table_metadata( location: str, properties: Properties = EMPTY_DICT, table_uuid

Re: [PR] fix `KeyError` raised by `add_files` when parquet file doe not have column stats [iceberg-python]

2024-11-23 Thread via GitHub
kevinjqliu commented on code in PR #1354: URL: https://github.com/apache/iceberg-python/pull/1354#discussion_r1855235419 ## tests/io/test_pyarrow_stats.py: ## @@ -681,6 +685,39 @@ def test_stats_types(table_schema_nested: Schema) -> None: ] +def test_read_missing_statis

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-11-23 Thread via GitHub
twuebi commented on code in PR #11317: URL: https://github.com/apache/iceberg/pull/11317#discussion_r1855212216 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]

2024-11-23 Thread via GitHub
kevinjqliu commented on issue #1363: URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2495556035 Thanks @JasperHG90 for the breakdown. The server spec above describes what the server should send and what the client should expect. According to the spec, Polaris shou

Re: [PR] add assertions in TestRowDelta [iceberg]

2024-11-23 Thread via GitHub
sullis closed pull request #11594: add assertions in TestRowDelta URL: https://github.com/apache/iceberg/pull/11594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] add assertions in TestRowDelta [iceberg]

2024-11-23 Thread via GitHub
sullis commented on code in PR #11594: URL: https://github.com/apache/iceberg/pull/11594#discussion_r1855223900 ## core/src/test/java/org/apache/iceberg/TestRowDelta.java: ## @@ -74,6 +74,9 @@ public void addOnlyDeleteFilesProducesDeleteOperation() { assertThat(snap.sequenc

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

2024-11-23 Thread via GitHub
twuebi commented on code in PR #11317: URL: https://github.com/apache/iceberg/pull/11317#discussion_r1855212216 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-23 Thread via GitHub
manuzhang commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855205570 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||-