Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
liurenjie1024 commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423583040 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -30,6 +30,9 @@ use self::{ use super::{FormatVersion, StructType}; +/// The seq number when no added file

Re: [PR] Hive: Use jUnit5 based HiveMetastoreExtension with Hive catalog tests [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9282: URL: https://github.com/apache/iceberg/pull/9282#discussion_r1423567661 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveTableBaseTest.java: ## @@ -26,22 +26,34 @@ import java.io.File; import java.nio.file.Paths; import java.ut

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
liurenjie1024 commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423579372 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1847 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] Hive: Use jUnit5 based HiveMetastoreExtension with Hive catalog tests [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9282: URL: https://github.com/apache/iceberg/pull/9282#discussion_r1423562265 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -288,7 +288,7 @@ public void createNamespace(Namespace namespace, Map meta) { } c

Re: [PR] Core, Hive, Nessie: Use ResolvingFileIO as default instead of HadoopFileIO [iceberg]

2023-12-11 Thread via GitHub
nastra commented on PR #8272: URL: https://github.com/apache/iceberg/pull/8272#issuecomment-1851448402 @ajantha-bhat this will be done prior to releasing 2.0.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1423554194 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Serializabl

Re: [PR] Core: Use properties while initializing default HadoopFileIO for Hadoop catalog. [iceberg]

2023-12-11 Thread via GitHub
agrawalreetika commented on PR #9283: URL: https://github.com/apache/iceberg/pull/9283#issuecomment-1851439389 Thank you @ajantha-bhat for the details. I have made the changes in the Title and description. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Release: 1.4.1 Javadocs [iceberg-docs]

2023-12-11 Thread via GitHub
Fokko closed pull request #289: Release: 1.4.1 Javadocs URL: https://github.com/apache/iceberg-docs/pull/289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423535271 ## crates/iceberg/src/spec/transform.rs: ## @@ -126,17 +126,20 @@ pub enum Transform { impl Transform { /// Get the return type of transform given the input type.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423537034 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1847 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423535271 ## crates/iceberg/src/spec/transform.rs: ## @@ -126,17 +126,20 @@ pub enum Transform { impl Transform { /// Get the return type of transform given the input type.

Re: [PR] Set HadoopFileIO as default for HadoopCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on PR #9283: URL: https://github.com/apache/iceberg/pull/9283#issuecomment-1851423776 @agrawalreetika: Previously also `HadoopFileIO` was default for HadoopCatalog. I think only problem was properties was not considered for the default `HadoopFileIO` initilaiz

Re: [PR] Core, Hive, Nessie: Use ResolvingFileIO as default instead of HadoopFileIO [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on PR #8272: URL: https://github.com/apache/iceberg/pull/8272#issuecomment-1851418956 I remember discussing this in the community sync. Why this didn't move forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-11 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1423526180 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -30,6 +30,9 @@ use self::{ use super::{FormatVersion, StructType}; +/// The seq number when no added files are pr

Re: [PR] Set HadoopFileIO as default for HadoopCatalog [iceberg]

2023-12-11 Thread via GitHub
agrawalreetika commented on PR #9283: URL: https://github.com/apache/iceberg/pull/9283#issuecomment-1851415341 @nastra, Could you please help me with review on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-11 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1423500567 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.a

[PR] Hive: Use jUnit5 based HiveMetastoreExtension with Hive catalog tests [iceberg]

2023-12-11 Thread via GitHub
nk1506 opened a new pull request, #9282: URL: https://github.com/apache/iceberg/pull/9282 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Data: Add GenericFileWriterFactory [iceberg]

2023-12-11 Thread via GitHub
szehon-ho commented on code in PR #9267: URL: https://github.com/apache/iceberg/pull/9267#discussion_r1423408972 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestMetadataTableReadableMetrics.java: ## @@ -219,27 +219,27 @@ public void testPrimitiveColumns()

Re: [PR] Data: Add GenericFileWriterFactory [iceberg]

2023-12-11 Thread via GitHub
szehon-ho commented on code in PR #9267: URL: https://github.com/apache/iceberg/pull/9267#discussion_r1423408972 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestMetadataTableReadableMetrics.java: ## @@ -219,27 +219,27 @@ public void testPrimitiveColumns()

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2023-12-11 Thread via GitHub
chandu-1101 commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1851283161 Abandon Iceberg, Abandon Hudi. Both are useless frameworks. Go with Parquet and spark sql. Your life will be simple and happy -- This is an automated message from the Apache Gi

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2023-12-11 Thread via GitHub
harshith-bolar-rapido commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1851252377 Any update on this? Facing the same issue with 1.2.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[I] Create Branches / TAGS between 2 snapshots [iceberg]

2023-12-11 Thread via GitHub
fanaticjo opened a new issue, #9281: URL: https://github.com/apache/iceberg/issues/9281 ### Feature Request / Improvement Is there a way where we can create a branch / TAG based on 2 snapshot ids or only latest data We have a use case where we write monthly generated report to

[I] Create Branches / TAGS between 2 snapshots [iceberg]

2023-12-11 Thread via GitHub
fanaticjo opened a new issue, #9280: URL: https://github.com/apache/iceberg/issues/9280 ### Feature Request / Improvement Is there a way where we can create a branch / TAG based on 2 snapshot ids or only latest data We have a use case where we write monthly generated report to

Re: [I] Documentation [iceberg-rust]

2023-12-11 Thread via GitHub
Xuanwo commented on issue #114: URL: https://github.com/apache/iceberg-rust/issues/114#issuecomment-1851232081 > Just curious, what would be the relationship with docs of crates on docs.rs? From the ASF perspective, anything outside of `apache.org` does not exist. We should host our

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1423322127 ## data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java: ## @@ -215,11 +215,11 @@ private static DataFile buildDataFile( .build(); }

Re: [I] Documentation [iceberg-rust]

2023-12-11 Thread via GitHub
liurenjie1024 commented on issue #114: URL: https://github.com/apache/iceberg-rust/issues/114#issuecomment-1851182139 cc @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spark 3.5: Add Spark application id to summary of RewriteDataFilesSparkAction [iceberg]

2023-12-11 Thread via GitHub
manuzhang commented on PR #9273: URL: https://github.com/apache/iceberg/pull/9273#issuecomment-1851176081 @amogh-jahagirdar @nastra could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
manuzhang commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1423305940 ## docs/spark-procedures.md: ## @@ -639,6 +639,7 @@ Keep in mind the `add_files` procedure will fetch the Parquet metadata from each | `source_table` | ✔️

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
manuzhang commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1423283454 ## data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java: ## @@ -215,11 +215,11 @@ private static DataFile buildDataFile( .build(); } - pr

Re: [I] DecimalType declaration check missing proper assertions [iceberg]

2023-12-11 Thread via GitHub
github-actions[bot] commented on issue #7420: URL: https://github.com/apache/iceberg/issues/7420#issuecomment-1851098029 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Inconsistent API for remove_orphan_files and DeleteOrphanFiles [iceberg]

2023-12-11 Thread via GitHub
github-actions[bot] commented on issue #7480: URL: https://github.com/apache/iceberg/issues/7480#issuecomment-1851098001 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Is is possible to control the number of partitions (groups) for compaction ? [iceberg]

2023-12-11 Thread via GitHub
github-actions[bot] commented on issue #7506: URL: https://github.com/apache/iceberg/issues/7506#issuecomment-1851097979 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Quickstart should give an example of a REST catalog enablement in the spark iceberg docker compose setup [iceberg]

2023-12-11 Thread via GitHub
github-actions[bot] closed issue #7615: Quickstart should give an example of a REST catalog enablement in the spark iceberg docker compose setup URL: https://github.com/apache/iceberg/issues/7615 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [I] Quickstart should give an example of a REST catalog enablement in the spark iceberg docker compose setup [iceberg]

2023-12-11 Thread via GitHub
github-actions[bot] commented on issue #7615: URL: https://github.com/apache/iceberg/issues/7615#issuecomment-1851097951 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1423222132 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.S

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1423238036 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -530,14 +515,16 @@ private static void importUnpartitionedSparkTable(

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on PR #9278: URL: https://github.com/apache/iceberg/pull/9278#issuecomment-1851053058 Ah I see the flink failure was fixed recently in https://github.com/apache/iceberg/pull/9216, I just need to rebase -- This is an automated message from the Apache Git Service.

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on PR #9278: URL: https://github.com/apache/iceberg/pull/9278#issuecomment-1851051133 Looks like some unrelated Flink tests failed, noting here: ``` TestIcebergSourceWithWatermarkExtractor > testThrottling FAILED java.lang.AssertionError:

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar closed pull request #9278: Core, API: Move SQLViewRepresentation to API URL: https://github.com/apache/iceberg/pull/9278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1423222132 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.S

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1423222132 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.S

[I] Querying metadata tables for a branch or tag [iceberg]

2023-12-11 Thread via GitHub
wypoon opened a new issue, #9279: URL: https://github.com/apache/iceberg/issues/9279 ### Query engine _No response_ ### Question The documentation states that metadata tables can be queried with `TIMESTAMP AS OF` and `VERSION AS OF`. As tables can be queried for a branch

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1423218929 ## core/src/main/java/org/apache/iceberg/view/BaseSQLViewRepresentation.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-11 Thread via GitHub
amogh-jahagirdar opened a new pull request, #9278: URL: https://github.com/apache/iceberg/pull/9278 This change is similar to the one done a while back for `BaseViewVersion`. SQLViewRepresentation is moved from the core module to the API module (since it is a concept that's apparent in the

[PR] Build: Bump typing-extensions from 4.8.0 to 4.9.0 [iceberg-python]

2023-12-11 Thread via GitHub
dependabot[bot] opened a new pull request, #204: URL: https://github.com/apache/iceberg-python/pull/204 Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.8.0 to 4.9.0. Release notes Sourced from https://github.com/python/typing_extensions/releases";>typi

[PR] Build: Bump mkdocs-material from 9.5.1 to 9.5.2 [iceberg-python]

2023-12-11 Thread via GitHub
dependabot[bot] opened a new pull request, #203: URL: https://github.com/apache/iceberg-python/pull/203 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.1 to 9.5.2. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdo

[PR] Core: Add StandardEncryptionManager [iceberg]

2023-12-11 Thread via GitHub
rdblue opened a new pull request, #9277: URL: https://github.com/apache/iceberg/pull/9277 This is an update to https://github.com/apache/iceberg/pull/6884 with a few final fixes to the `StandardEncryptionManager` and related classes. This is almost entirely @ggershinsky's work, I'm just sub

Re: [PR] Support parameterized tests at class-level with JUnit5 [iceberg]

2023-12-11 Thread via GitHub
GianlucaPrincipini commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1423154765 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -173,6 +178,60 @@ public static void assertSameSchemaMap(Map map1, Map

Re: [I] DatasourceV2 does not prune columns after V2ScanRelationPushDown [iceberg]

2023-12-11 Thread via GitHub
akshayakp97 commented on issue #9268: URL: https://github.com/apache/iceberg/issues/9268#issuecomment-1850822846 In general, if a `Project` is added after the execution of `V2ScanRelationPushDown` rule - how do the columns get pruned? Or, do we not expect any new `Project`'s? -- This is

Re: [I] DatasourceV2 does not prune columns after V2ScanRelationPushDown [iceberg]

2023-12-11 Thread via GitHub
akshayakp97 commented on issue #9268: URL: https://github.com/apache/iceberg/issues/9268#issuecomment-1850797730 After `ColumnPruning` adds the new `Project [cs_warehouse_sk#54840, cs_order_number#54843L]`, when `V2ScanRelationPushDown` rule triggers, it doesn't match the [`ScanOperation`]

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
singhpk234 commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1423038582 ## docs/spark-procedures.md: ## @@ -639,6 +639,7 @@ Keep in mind the `add_files` procedure will fetch the Parquet metadata from each | `source_table` | ✔️

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-12-11 Thread via GitHub
singhpk234 commented on PR #8980: URL: https://github.com/apache/iceberg/pull/8980#issuecomment-1850795022 > Can you explain what is the purpose of using existingFilesCount here ? I am not fully aware of this logically i totally agree with you it makes no sense to keep it but what i a

Re: [I] DatasourceV2 does not prune columns after V2ScanRelationPushDown [iceberg]

2023-12-11 Thread via GitHub
akshayakp97 commented on issue #9268: URL: https://github.com/apache/iceberg/issues/9268#issuecomment-1850779601 Thanks for your response. I am looking at TPCDS q16 physical plan for Iceberg on EMR. Link to q16 - https://github.com/apache/spark/blob/a78d6ce376edf2a8836e01f47b

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-11 Thread via GitHub
dramaticlly commented on PR #8340: URL: https://github.com/apache/iceberg/pull/8340#issuecomment-1850721026 @rdblue @amogh-jahagirdar can you take another look? I think it's awesome if we can merge this fix. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] DatasourceV2 does not prune columns after V2ScanRelationPushDown [iceberg]

2023-12-11 Thread via GitHub
rdblue commented on issue #9268: URL: https://github.com/apache/iceberg/issues/9268#issuecomment-1850720036 I don't think I'm following the logic here. Is there a case where you're not seeing columns being properly pruned? -- This is an automated message from the Apache Git Service. To re

Re: [PR] Add doc for rewriting manifest with spec id [iceberg]

2023-12-11 Thread via GitHub
puchengy commented on PR #9253: URL: https://github.com/apache/iceberg/pull/9253#issuecomment-1850509409 @aokolnychyi Addressed your comment, PTAL, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422818781 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected Map

[I] Support schema.name-mapping.default Column Projection property [iceberg-python]

2023-12-11 Thread via GitHub
syun64 opened a new issue, #202: URL: https://github.com/apache/iceberg-python/issues/202 ### Feature Request / Improvement schema.name-mapping.default property is well supported in Spark Iceberg in order to enable column consistent reads for Iceberg tables that rely on migration pro

Re: [PR] Add SQLite support [iceberg-python]

2023-12-11 Thread via GitHub
rdblue merged PR #178: URL: https://github.com/apache/iceberg-python/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-11 Thread via GitHub
pvary commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1422793318 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier originalT

Re: [PR] Support parameterized tests at class-level with JUnit5 [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1422771246 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -173,6 +178,60 @@ public static void assertSameSchemaMap(Map map1, Map

Re: [PR] Support parameterized tests at class-level with JUnit5 [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1422771246 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -173,6 +178,60 @@ public static void assertSameSchemaMap(Map map1, Map

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1422755860 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.a

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1422755860 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreTest.java: ## @@ -31,6 +31,10 @@ import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.a

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1422744838 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1422744205 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Table Requirements Validation [iceberg-python]

2023-12-11 Thread via GitHub
Fokko merged PR #200: URL: https://github.com/apache/iceberg-python/pull/200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422711943 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -448,33 +480,85 @@ public void renameTable(TableIdentifier from, TableIdentifier

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422711613 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -133,69 +114,30 @@ protected void doCommit(TableMetadata base, TableMetadata m

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422710861 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -133,69 +114,30 @@ protected void doCommit(TableMetadata base, TableMetadata m

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422706464 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected Map p

Re: [I] java.lang.ClassNotFoundException: Failed to find data source: iceberg. Issue when we are using Java Custom Catalog [iceberg]

2023-12-11 Thread via GitHub
mahendrachandrasekhar commented on issue #9275: URL: https://github.com/apache/iceberg/issues/9275#issuecomment-1850354505 We have this included in our docker image (which is based of bitnami/spark:3.2.4 RUN curl -L -o /home/airflow/spark/jars/hadoop-aws.jar https://repo.maven.apache

[I] java.lang.ClassNotFoundException: Failed to find data source: iceberg. Issue when we are using Java Custom Catalog [iceberg]

2023-12-11 Thread via GitHub
mahendrachandrasekhar opened a new issue, #9275: URL: https://github.com/apache/iceberg/issues/9275 We are using a Java Custom Catalog with iceberg. The Table is created properly, however we get an issue when we insert the data. `public String createCustomTable(String tableName) {

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422658039 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -133,69 +114,30 @@ protected void doCommit(TableMetadata base, TableMetadata metada

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422656154 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieViewCatalog.java: ## @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422655391 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422637569 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected Map

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422637569 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected Map

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422634486 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -133,69 +114,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadat

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422626565 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieViewCatalog.java: ## @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422622180 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

[PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-11 Thread via GitHub
manuzhang opened a new pull request, #9274: URL: https://github.com/apache/iceberg/pull/9274 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422564943 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -133,69 +114,30 @@ protected void doCommit(TableMetadata base, TableMetadata metada

Re: [PR] Api: Track partition statistics via TableMetadata [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8502: URL: https://github.com/apache/iceberg/pull/8502#discussion_r1422543077 ## .palantir/revapi.yml: ## @@ -873,6 +873,10 @@ acceptedBreaks: new: "method void org.apache.iceberg.encryption.Ciphers::()" justification: "Static

Re: [PR] Api: Track partition statistics via TableMetadata [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8502: URL: https://github.com/apache/iceberg/pull/8502#discussion_r1422541957 ## core/src/main/java/org/apache/iceberg/ReachableFileUtil.java: ## @@ -137,7 +137,9 @@ public static List manifestListLocations(Table table, Set snapshot *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422535596 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422521681 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422508540 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -347,4 +339,65 @@ private TableIdentifier identifierWithoutTableReference( protected Map

Re: [I] iceberg-spark: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-11 Thread via GitHub
chinmay-bhat commented on issue #9086: URL: https://github.com/apache/iceberg/issues/9086#issuecomment-1850148369 As it's a big task, I'll get started migrating tests in `iceberg-spark` v3.5 that are not parameterized, and later open a new PR for the parameterized ones :) -- This is an a

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1422497424 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

[PR] Spark 3.5: Add Spark application id to summary of RewriteDataFilesSparkAction [iceberg]

2023-12-11 Thread via GitHub
manuzhang opened a new pull request, #9273: URL: https://github.com/apache/iceberg/pull/9273 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on PR #6887: URL: https://github.com/apache/iceberg/pull/6887#issuecomment-1849920644 PR is ready @danielcweeks, @nastra, @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9234: URL: https://github.com/apache/iceberg/pull/9234#discussion_r1422341041 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1384,7 +1386,9 @@ public Builder setPreviousFileLocation(String previousFileLocation) { private

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9234: URL: https://github.com/apache/iceberg/pull/9234#discussion_r1422342150 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1384,7 +1386,9 @@ public Builder setPreviousFileLocation(String previousFileLocation) { private

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9234: URL: https://github.com/apache/iceberg/pull/9234#discussion_r1422342150 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1384,7 +1386,9 @@ public Builder setPreviousFileLocation(String previousFileLocation) { private

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2023-12-11 Thread via GitHub
nastra commented on code in PR #9234: URL: https://github.com/apache/iceberg/pull/9234#discussion_r1422341041 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1384,7 +1386,9 @@ public Builder setPreviousFileLocation(String previousFileLocation) { private

Re: [I] iceberg-spark: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-11 Thread via GitHub
nastra commented on issue #9086: URL: https://github.com/apache/iceberg/issues/9086#issuecomment-1849884128 We should be mostly focusing on migrating Spark 3.5, which is already a big task -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] iceberg-spark: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-11 Thread via GitHub
chinmay-bhat commented on issue #9086: URL: https://github.com/apache/iceberg/issues/9086#issuecomment-1849868289 Ok, so can I pick this up once the parameterized tests PR is merged? Also, iceberg-spark has folders for each version (v3.2, 3.3, 3.4, 3.5). Do you recommend creating sep

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

2023-12-11 Thread via GitHub
ajantha-bhat commented on code in PR #6887: URL: https://github.com/apache/iceberg/pull/6887#discussion_r1422303733 ## core/src/main/java/org/apache/iceberg/avro/AvroFileAppender.java: ## @@ -78,14 +78,13 @@ public Metrics metrics() { @Override public long length() { -

Re: [PR] Data: Add GenericFileWriterFactory [iceberg]

2023-12-11 Thread via GitHub
aokolnychyi commented on code in PR #9267: URL: https://github.com/apache/iceberg/pull/9267#discussion_r1422256919 ## data/src/test/java/org/apache/iceberg/io/TestFileWriterFactory.java: ## @@ -76,18 +76,15 @@ public static Object[] parameters() { private final FileFormat f

  1   2   >