Re: [PR] Build: Bump org.assertj:assertj-core from 3.25.2 to 3.25.3 [iceberg]

2024-02-11 Thread via GitHub
nastra commented on PR #9706: URL: https://github.com/apache/iceberg/pull/9706#issuecomment-1938170342 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485815887 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java: ## @@ -150,33 +150,31 @@ public static Collection parameters() { privat

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485814843 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java: ## @@ -779,38 +771,35 @@ public void testIcebergAndHmsTableProperties() throws

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485813599 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java: ## @@ -140,35 +138,34 @@ public static Collection parameters() { private Te

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485812902 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerLocalScan.java: ## @@ -135,13 +135,13 @@ public void testScanTable() throws IOException {

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485813432 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java: ## @@ -61,21 +63,17 @@ import org.apache.iceberg.types.Type; import org.apache

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485812057 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerLocalScan.java: ## @@ -83,47 +83,47 @@ public static Collection parameters() { private T

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485811808 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithMultipleCatalogs.java: ## @@ -18,60 +18,63 @@ */ package org.apache.iceberg.mr.hive;

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485810742 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerTimezone.java: ## @@ -82,22 +82,21 @@ public static Collection parameters() { private Te

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485810072 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestTables.java: ## @@ -57,14 +60,12 @@ import org.apache.iceberg.relocated.com.google.common.base.Preconditions; import

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485810348 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestTables.java: ## @@ -417,18 +418,17 @@ static class CustomCatalogTestTables extends TestTables { private final S

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485809141 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -402,10 +402,11 @@ public List getSplits() { } public void validate(List expect

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485808992 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -336,26 +336,26 @@ public void testSnapshotReads() throws Exception { testInputFormat.

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485808472 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -237,8 +237,8 @@ public void testProjection() throws Exception { List outputRecords

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485808098 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -90,52 +93,49 @@ public class TestIcebergInputFormats { private static final PartitionSp

Re: [PR] fix #9269 Switch class level parameterized for mr [iceberg]

2024-02-11 Thread via GitHub
nastra commented on code in PR #9711: URL: https://github.com/apache/iceberg/pull/9711#discussion_r1485806497 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -90,52 +93,49 @@ public class TestIcebergInputFormats { private static final PartitionSp

Re: [PR] chore(deps): Update opendal requirement from 0.44 to 0.45 [iceberg-rust]

2024-02-11 Thread via GitHub
Fokko merged PR #195: URL: https://github.com/apache/iceberg-rust/pull/195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] Apache Project website check is failing [iceberg]

2024-02-11 Thread via GitHub
Fokko opened a new issue, #9712: URL: https://github.com/apache/iceberg/issues/9712 ### Feature Request / Improvement This is probably caused by the recent migration to the new framework. We should make sure that they are all passing: https://whimsy.apache.org/site/project/iceb

Re: [I] add support for DuckDB views as a valid data format [iceberg-python]

2024-02-11 Thread via GitHub
djouallah commented on issue #407: URL: https://github.com/apache/iceberg-python/issues/407#issuecomment-1938101894 here is a reproducible example, in real life I loaded 2100 files which generate an OOM. https://colab.research.google.com/drive/1yultO4Wc94kJMhnTzwa5Gcrfp8VeDubg#scrollTo=T

Re: [I] Reordering partition columns in iceberg spark [iceberg]

2024-02-11 Thread via GitHub
amogh-jahagirdar closed issue #9698: Reordering partition columns in iceberg spark URL: https://github.com/apache/iceberg/issues/9698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Reordering partition columns in iceberg spark [iceberg]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on issue #9698: URL: https://github.com/apache/iceberg/issues/9698#issuecomment-1938098558 Checkout https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table for how to re-order columns. >but it looks like rewriteManifests does not rewrite the metadata t

Re: [I] add support for DuckDB views as a valid data format [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on issue #407: URL: https://github.com/apache/iceberg-python/issues/407#issuecomment-1938083316 @djouallah Thanks for reporting, could you provide some more specifics on the tests you are running which are exceeding memory? If you're able to share queries, the sca

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485749975 ## pyiceberg/table/metadata.py: ## @@ -313,6 +315,34 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]: return dat

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485749975 ## pyiceberg/table/metadata.py: ## @@ -313,6 +315,34 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]: return dat

Re: [PR] Use the correct name of extra that adds GCS support [iceberg-python]

2024-02-11 Thread via GitHub
HonahX merged PR #412: URL: https://github.com/apache/iceberg-python/pull/412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485746204 ## pyiceberg/table/metadata.py: ## @@ -404,13 +406,34 @@ def new_table_metadata( properties: Properties = EMPTY_DICT, table_uuid: Optional[uuid.

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on PR #411: URL: https://github.com/apache/iceberg-python/pull/411#issuecomment-1938053112 >Thanks for taking this! @amogh-jahagirdar It's great that we no longer require extra before validator for supporting the v1 metadata. Setting the relevant fields explicitly

Re: [PR] Use the correct name of extra that adds GCS support [iceberg-python]

2024-02-11 Thread via GitHub
HonahX commented on code in PR #412: URL: https://github.com/apache/iceberg-python/pull/412#discussion_r1485743360 ## mkdocs/docs/index.md: ## @@ -55,9 +55,9 @@ You can mix and match optional dependencies depending on your needs: | s3fs | S3FS as a FileIO implementatio

Re: [PR] Use the correct name of extra that adds GCS support [iceberg-python]

2024-02-11 Thread via GitHub
HonahX commented on code in PR #412: URL: https://github.com/apache/iceberg-python/pull/412#discussion_r1485743195 ## mkdocs/docs/index.md: ## @@ -55,9 +55,9 @@ You can mix and match optional dependencies depending on your needs: | s3fs | S3FS as a FileIO implementatio

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485741811 ## pyiceberg/table/metadata.py: ## @@ -404,13 +406,34 @@ def new_table_metadata( properties: Properties = EMPTY_DICT, table_uuid: Optional[uuid.

Re: [PR] Use the correct name of extra that adds GCS support [iceberg-python]

2024-02-11 Thread via GitHub
HonahX commented on PR #412: URL: https://github.com/apache/iceberg-python/pull/412#issuecomment-1938034050 Great catch! Thanks for fixing this! @vaultah -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
HonahX commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485730495 ## pyiceberg/table/metadata.py: ## @@ -404,13 +406,34 @@ def new_table_metadata( properties: Properties = EMPTY_DICT, table_uuid: Optional[uuid.UUID] = No

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
HonahX commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485730495 ## pyiceberg/table/metadata.py: ## @@ -404,13 +406,34 @@ def new_table_metadata( properties: Properties = EMPTY_DICT, table_uuid: Optional[uuid.UUID] = No

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485700731 ## pyiceberg/table/metadata.py: ## @@ -411,6 +442,24 @@ def new_table_metadata( if table_uuid is None: table_uuid = uuid.uuid4() +# Re

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

2024-02-11 Thread via GitHub
rustyconover commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-1937932075 Hi @aokolnychyi can we change it to be a buffered binary writer that way we would get the length counts written? -- This is an automated message from the Apache Git Service. To res

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485696929 ## pyiceberg/table/metadata.py: ## @@ -313,6 +315,34 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]: return dat

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485695745 ## pyiceberg/table/metadata.py: ## @@ -411,6 +442,24 @@ def new_table_metadata( if table_uuid is None: table_uuid = uuid.uuid4() +# Re

Re: [I] Missing Types.UUIDType in SUPPORTED_PRIMITIVES [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] commented on issue #1302: URL: https://github.com/apache/iceberg/issues/1302#issuecomment-1937925323 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type UnusedVariable. [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] commented on issue #1291: URL: https://github.com/apache/iceberg/issues/1291#issuecomment-1937925287 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type NarrowingCompoundAssignment. [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] commented on issue #1290: URL: https://github.com/apache/iceberg/issues/1290#issuecomment-1937925272 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Suggestion for newbie getting started guide [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] commented on issue #761: URL: https://github.com/apache/iceberg/issues/761#issuecomment-1937925168 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Should we consider adding a github action to lint YAML files for correctness / proper style? [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] commented on issue #1300: URL: https://github.com/apache/iceberg/issues/1300#issuecomment-1937925303 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Suggestion for newbie getting started guide [iceberg]

2024-02-11 Thread via GitHub
github-actions[bot] closed issue #761: Suggestion for newbie getting started guide URL: https://github.com/apache/iceberg/issues/761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Build: Bump io.airlift:aircompressor from 0.25 to 0.26 [iceberg]

2024-02-11 Thread via GitHub
Fokko merged PR #9700: URL: https://github.com/apache/iceberg/pull/9700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Build: Bump org.testcontainers:testcontainers from 1.19.4 to 1.19.5 [iceberg]

2024-02-11 Thread via GitHub
Fokko merged PR #9704: URL: https://github.com/apache/iceberg/pull/9704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Build: Bump mkdocs-material from 9.5.7 to 9.5.9 [iceberg]

2024-02-11 Thread via GitHub
Fokko merged PR #9708: URL: https://github.com/apache/iceberg/pull/9708 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Docs: Fix broken strike-through markup [iceberg]

2024-02-11 Thread via GitHub
Fokko merged PR #9696: URL: https://github.com/apache/iceberg/pull/9696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] Strikethrough is broken in the specification [iceberg]

2024-02-11 Thread via GitHub
Fokko closed issue #9693: Strikethrough is broken in the specification URL: https://github.com/apache/iceberg/issues/9693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
Fokko commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485647073 ## pyiceberg/table/metadata.py: ## @@ -260,8 +260,10 @@ def set_v2_compatible_defaults(cls, data: Dict[str, Any]) -> Dict[str, Any]: The TableMetadata

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485645103 ## tests/catalog/test_hive.py: ## @@ -294,6 +294,37 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, assert meta

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485645020 ## pyiceberg/table/metadata.py: ## @@ -313,6 +315,34 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]: return dat

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

2024-02-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485627063 ## pyiceberg/table/metadata.py: ## @@ -313,6 +315,34 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]: return dat

Re: [PR] Support for pushdown like filter (endsWith and contains) [iceberg]

2024-02-11 Thread via GitHub
yabola commented on PR #9683: URL: https://github.com/apache/iceberg/pull/9683#issuecomment-1937761335 > Also, I'll need to think more if we can actually support this for delete files. If not, this will need to only be applied for CoW tables. For example, for agg pushdown, we don't support

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-11 Thread via GitHub
ggershinsky commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1483931060 ## core/src/main/java/org/apache/iceberg/ManifestFiles.java: ## @@ -345,34 +345,24 @@ private static ManifestFile copyManifestInternal( return writer.toManifes

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-11 Thread via GitHub
ggershinsky commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1485596071 ## core/src/main/java/org/apache/iceberg/ManifestFiles.java: ## @@ -345,34 +345,24 @@ private static ManifestFile copyManifestInternal( return writer.toManifes

[PR] chore(deps): Update opendal requirement from 0.44 to 0.45 [iceberg-rust]

2024-02-11 Thread via GitHub
dependabot[bot] opened a new pull request, #195: URL: https://github.com/apache/iceberg-rust/pull/195 Updates the requirements on [opendal](https://github.com/apache/opendal) to permit the latest version. Release notes Sourced from https://github.com/apache/opendal/releases";>opend

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-02-11 Thread via GitHub
ilyasahsan123 closed pull request #9565: Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) URL: https://github.com/apache/iceberg/pull/9565 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add filter pushdown filter API for contains and endsWith [iceberg]

2024-02-11 Thread via GitHub
yabola commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-1937485214 The origin PR is https://github.com/apache/iceberg/pull/9683 , I split it into two PRs and add some UT. -- This is an automated message from the Apache Git Service. To respond to the mes