[I] Can databricks volume be implemented on Iceberg? [iceberg]

2023-12-07 Thread via GitHub
coolderli opened a new issue, #9249: URL: https://github.com/apache/iceberg/issues/9249 ### Query engine _No response_ ### Question Recently, I was researching solutions for managing unstructured files and discovered [the volume of databricks](https://docs.databrick

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-07 Thread via GitHub
liurenjie1024 commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1420028918 ## crates/iceberg/src/spec/transform.rs: ## @@ -126,17 +126,20 @@ pub enum Transform { impl Transform { /// Get the return type of transform given the inpu

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-07 Thread via GitHub
liurenjie1024 commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1420025262 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -30,6 +30,9 @@ use self::{ use super::{FormatVersion, StructType}; +/// The seq number when no added file

Re: [I] How to connect apache iceberg to minio [iceberg]

2023-12-07 Thread via GitHub
selsebil82 commented on issue #9205: URL: https://github.com/apache/iceberg/issues/9205#issuecomment-1846660737 hey, did you find a solution for your problem ? because am also having the same one -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1420006288 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieViewCatalog.java: ## @@ -0,0 +1,217 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [I] Duplicate file name in Iceberg's metadata [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on issue #8953: URL: https://github.com/apache/iceberg/issues/8953#issuecomment-1846634916 I also want to verify why fileCount doesn't really cover the uniqueness right now, the only other way would be if it's in a different thread (and both threads just end up ha

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419995071 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -267,4 +305,23 @@ static DataFile makeDataFile(Table icebergTable, String fileLocati

Re: [I] Duplicate file name in Iceberg's metadata [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on issue #8953: URL: https://github.com/apache/iceberg/issues/8953#issuecomment-1846628756 Ok I actually looked at the history of these changes now https://github.com/apache/iceberg/pull/5214 was never merged but followed by https://github.com/apache/iceberg/pull/

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419970848 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -141,16 +147,26 @@ private UpdateableReference loadReference(String requestedRef

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419953116 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -141,16 +147,26 @@ private UpdateableReference loadReference(String requestedRef

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-07 Thread via GitHub
Xuanwo commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1419945979 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1847 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Arrow: Allow missing field-ids from Schema [iceberg-python]

2023-12-07 Thread via GitHub
HonahX commented on code in PR #183: URL: https://github.com/apache/iceberg-python/pull/183#discussion_r1419923194 ## pyiceberg/io/pyarrow.py: ## @@ -713,28 +714,50 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: """Visit a primitive type.""" -def _g

Re: [PR] Arrow: Allow missing field-ids from Schema [iceberg-python]

2023-12-07 Thread via GitHub
HonahX commented on code in PR #183: URL: https://github.com/apache/iceberg-python/pull/183#discussion_r1419912292 ## pyiceberg/io/pyarrow.py: ## @@ -713,28 +721,49 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: """Visit a primitive type.""" -def _g

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-07 Thread via GitHub
ZENOTME commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1419902834 ## crates/iceberg/src/spec/transform.rs: ## @@ -126,17 +126,20 @@ pub enum Transform { impl Transform { /// Get the return type of transform given the input type

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-07 Thread via GitHub
ZENOTME commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1419901970 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -30,6 +30,9 @@ use self::{ use super::{FormatVersion, StructType}; +/// The seq number when no added files are

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-07 Thread via GitHub
ZENOTME commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1419881701 ## crates/catalog/rest/src/catalog.rs: ## @@ -163,15 +163,45 @@ impl HttpClient { ) -> Result<()> { let resp = self.0.execute(request).await?; +

Re: [PR] Spark: IN clause on system function is not pushed down [iceberg]

2023-12-07 Thread via GitHub
advancedxy commented on code in PR #9192: URL: https://github.com/apache/iceberg/pull/9192#discussion_r1419869752 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala: ## @@ -40,14 +37,20 @@ import org.apache.spark.sql.t

Re: [I] Timestamp data losing precision when writing with Flink [iceberg]

2023-12-07 Thread via GitHub
yeunghl-shoalter closed issue #9229: Timestamp data losing precision when writing with Flink URL: https://github.com/apache/iceberg/issues/9229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Timestamp data losing precision when writing with Flink [iceberg]

2023-12-07 Thread via GitHub
yeunghl-shoalter commented on issue #9229: URL: https://github.com/apache/iceberg/issues/9229#issuecomment-1846456239 It is found that Flink CDC MySQL can only provide event with source operation timestamp in precision of second. This is not an issue of converting Flink data into Iceberg da

Re: [PR] Core: Fix null partitions in PartitionSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9248: URL: https://github.com/apache/iceberg/pull/9248#discussion_r1419859782 ## core/src/test/java/org/apache/iceberg/util/TestPartitionMap.java: ## @@ -274,9 +274,11 @@ public void testKeyAndEntrySetEquality() { map1.put(BY_DATA_SPEC

Re: [PR] Core: Fix null partitions in PartitionSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9248: URL: https://github.com/apache/iceberg/pull/9248#discussion_r1419859485 ## core/src/test/java/org/apache/iceberg/util/TestPartitionSet.java: ## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[PR] Core: Fix null partitions in PartitionSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi opened a new pull request, #9248: URL: https://github.com/apache/iceberg/pull/9248 This PR fixes usage of `null` partitions in `PartitionSet`. Previously, it was possible to store null partitions for unpartitioned specs but we could not retrieve them back. The set returned `fals

Re: [PR] feat: support UnboundPartitionSpec [iceberg-rust]

2023-12-07 Thread via GitHub
liurenjie1024 commented on code in PR #106: URL: https://github.com/apache/iceberg-rust/pull/106#discussion_r1419858086 ## crates/iceberg/src/catalog/mod.rs: ## @@ -758,23 +760,19 @@ mod tests { { "action": "add-spec", "spec": { -"spec-id": 1, "fields

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419845857 ## api/src/test/java/org/apache/iceberg/util/TestCharSequenceSet.java: ## @@ -79,4 +81,35 @@ public void testRemoveAll() { Assertions.assertThat(set).isEmpty

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419844248 ## api/src/main/java/org/apache/iceberg/util/CharSequenceSet.java: ## @@ -168,22 +167,29 @@ public void clear() { } @Override - public boolean equals(Objec

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419845126 ## api/src/main/java/org/apache/iceberg/util/CharSequenceSet.java: ## @@ -168,22 +167,29 @@ public void clear() { } @Override - public boolean equals(Objec

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419844796 ## api/src/main/java/org/apache/iceberg/util/CharSequenceSet.java: ## @@ -168,22 +167,29 @@ public void clear() { } @Override - public boolean equals(Objec

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419844248 ## api/src/main/java/org/apache/iceberg/util/CharSequenceSet.java: ## @@ -168,22 +167,29 @@ public void clear() { } @Override - public boolean equals(Objec

Re: [I] org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore at [iceberg]

2023-12-07 Thread via GitHub
whymed commented on issue #9030: URL: https://github.com/apache/iceberg/issues/9030#issuecomment-1846330182 Hi, I was not able to fix the issue... I ended ditching hive/iceberg and doing ETL using R/sparklyr/spark and L directly on the same AWS RDS instance but in a diferent DBs from the

Re: [I] Create namespace through the CLI [iceberg-python]

2023-12-07 Thread via GitHub
detunjiSamuel commented on issue #185: URL: https://github.com/apache/iceberg-python/issues/185#issuecomment-1846301788 hey, I would love to take up this task. Please could you assign it to me? -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] API, Core: Add sqlFor API to views to handle basic resolution of dialect [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #9247: URL: https://github.com/apache/iceberg/pull/9247#discussion_r1419760735 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -200,6 +200,13 @@ public void completeCreateView() { .build

Re: [PR] API, Core: Add sqlFor API to views to handle basic resolution of dialect [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #9247: URL: https://github.com/apache/iceberg/pull/9247#discussion_r1419759456 ## api/src/main/java/org/apache/iceberg/view/View.java: ## @@ -121,4 +121,15 @@ default UpdateLocation updateLocation() { default UUID uuid() { throw n

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

2023-12-07 Thread via GitHub
stevenzwu commented on code in PR #9173: URL: https://github.com/apache/iceberg/pull/9173#discussion_r1419756211 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -105,12 +107,12 @@ public class IcebergSource implements Source

Re: [PR] Flink: switch to use SortKey for data statistics [iceberg]

2023-12-07 Thread via GitHub
stevenzwu commented on code in PR #9212: URL: https://github.com/apache/iceberg/pull/9212#discussion_r1419739020 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestDataStatisticsOperator.java: ## @@ -119,9 +121,9 @@ public void testProcessElement() thro

[PR] Build: Bump mkdocs-material from 9.4.14 to 9.5.0 [iceberg-python]

2023-12-07 Thread via GitHub
dependabot[bot] opened a new pull request, #196: URL: https://github.com/apache/iceberg-python/pull/196 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.4.14 to 9.5.0. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkd

[I] DeltaLakeToIcebergMigration [iceberg]

2023-12-07 Thread via GitHub
arifazmidd opened a new issue, #9246: URL: https://github.com/apache/iceberg/issues/9246 ### Query engine _No response_ ### Question I'm using [DeltaLakeToIcebergMigrationActionsProvider](https://iceberg.apache.org/docs/1.3.0/delta-lake-migration/) and am seeing it take

[PR] Build: Bump actions/stale from 8.0.0 to 9.0.0 [iceberg-python]

2023-12-07 Thread via GitHub
dependabot[bot] opened a new pull request, #195: URL: https://github.com/apache/iceberg-python/pull/195 Bumps [actions/stale](https://github.com/actions/stale) from 8.0.0 to 9.0.0. Release notes Sourced from https://github.com/actions/stale/releases";>actions/stale's releases.

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-12-07 Thread via GitHub
rdblue commented on PR #61: URL: https://github.com/apache/iceberg-python/pull/61#issuecomment-1846157302 Thanks, @Fokko! Looks great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-12-07 Thread via GitHub
rdblue merged PR #61: URL: https://github.com/apache/iceberg-python/pull/61 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-12-07 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1419692800 ## pyiceberg/table/snapshots.py: ## @@ -65,6 +90,25 @@ def __init__(self, operation: Operation, **data: Any) -> None: super().__init__(operation=operation,

Re: [PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi commented on code in PR #9245: URL: https://github.com/apache/iceberg/pull/9245#discussion_r1419690269 ## api/src/test/java/org/apache/iceberg/util/TestCharSequenceSet.java: ## @@ -79,4 +81,35 @@ public void testRemoveAll() { Assertions.assertThat(set).isEmpty

[PR] API: Fix equals and hashCode behavior in CharSequenceSet [iceberg]

2023-12-07 Thread via GitHub
aokolnychyi opened a new pull request, #9245: URL: https://github.com/apache/iceberg/pull/9245 The `equals` and `hashCode` behaviors contradict the `Set` API, which prohibits wrapping `CharSequenceSet` instances into unmodifiable wrappers in `CharSequenceMap$keySet`. ``` /** *

Re: [PR] [spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
RussellSpitzer commented on PR #9242: URL: https://github.com/apache/iceberg/pull/9242#issuecomment-1846146418 Thanks @puchengy for the pr! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] [spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
RussellSpitzer merged PR #9242: URL: https://github.com/apache/iceberg/pull/9242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [I] Unable to create iceberg database / table using spark sql with AWS S3 + Glue integration [iceberg]

2023-12-07 Thread via GitHub
ExplorData24 commented on issue #3131: URL: https://github.com/apache/iceberg/issues/3131#issuecomment-1846120664 @jackye1995 @qq240035000 @bvinayakumar Hello, Almost I have the same problem when I tried to execute this script: import os import pyspark from pyspark

Re: [PR] Flink: switch to use SortKey for data statistics [iceberg]

2023-12-07 Thread via GitHub
pvary commented on code in PR #9212: URL: https://github.com/apache/iceberg/pull/9212#discussion_r1419662475 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestDataStatisticsOperator.java: ## @@ -119,9 +121,9 @@ public void testProcessElement() throws E

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-07 Thread via GitHub
jasonf20 commented on code in PR #9230: URL: https://github.com/apache/iceberg/pull/9230#discussion_r1419551515 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -313,6 +313,37 @@ public void testRecoveryWithoutManifestList() { metadata.currentSnapsho

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-12-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1419654637 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +269,162 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

2023-12-07 Thread via GitHub
pvary commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1419643002 ## data/src/test/java/org/apache/iceberg/data/GenAppenderHelper.java: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-07 Thread via GitHub
jayceslesar opened a new pull request, #194: URL: https://github.com/apache/iceberg-python/pull/194 Spawned from #156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Multiple Catalog Connection [iceberg-python]

2023-12-07 Thread via GitHub
jayceslesar commented on issue #156: URL: https://github.com/apache/iceberg-python/issues/156#issuecomment-1846064006 Im happy to add this to the docs, wondering the best place for that? With mkdocs it seems to be the best place to add docs would be in docstrings. Thoughts? Ideally there ar

[I] Unable to create Iceberg table with HiveCatalog: org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore [iceberg]

2023-12-07 Thread via GitHub
ExplorData24 opened a new issue, #9244: URL: https://github.com/apache/iceberg/issues/9244 ### Apache Iceberg version 1.0.0 ### Query engine Spark ### Please describe the bug 🐞 Hello everyone. I am using Hive Catalog to create Iceberg tables with Spar

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-07 Thread via GitHub
jasonf20 commented on code in PR #9230: URL: https://github.com/apache/iceberg/pull/9230#discussion_r1419551515 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -313,6 +313,37 @@ public void testRecoveryWithoutManifestList() { metadata.currentSnapsho

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-07 Thread via GitHub
jasonf20 commented on code in PR #9230: URL: https://github.com/apache/iceberg/pull/9230#discussion_r1419551515 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -313,6 +313,37 @@ public void testRecoveryWithoutManifestList() { metadata.currentSnapsho

Re: [I] org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore at [iceberg]

2023-12-07 Thread via GitHub
ExplorData24 commented on issue #9030: URL: https://github.com/apache/iceberg/issues/9030#issuecomment-1845986735 @whymed @zhangbutao @zyx199693 Hello, Good morning, Almost I have the same problem when I tried to execute the query: spark.sql("CREATE TABLE catalog_hi

Re: [PR] Delta: Fix integration tests and Create DataFile by partition values instead of path [iceberg]

2023-12-07 Thread via GitHub
jackye1995 merged PR #8398: URL: https://github.com/apache/iceberg/pull/8398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Delta: Fix integration tests and Create DataFile by partition values instead of path [iceberg]

2023-12-07 Thread via GitHub
jackye1995 commented on PR #8398: URL: https://github.com/apache/iceberg/pull/8398#issuecomment-1845981450 CI seems to be passing now after retry. Given the fact that this issue was breaking CI, I will go ahead to merge it directly. Thanks for the fix! -- This is an automated message fro

Re: [I] Support bulk remove orphan files [iceberg]

2023-12-07 Thread via GitHub
jackye1995 commented on issue #7111: URL: https://github.com/apache/iceberg/issues/7111#issuecomment-1845978489 Related devlist discussion: https://lists.apache.org/thread/9swm4jo8jjlj5q2jxct5cwo4yv2m8tcd -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

2023-12-07 Thread via GitHub
stevenzwu commented on PR #9211: URL: https://github.com/apache/iceberg/pull/9211#issuecomment-1845955767 thanks @rodmeneses for the contribution and @pvary for following up with the Flink community -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

2023-12-07 Thread via GitHub
stevenzwu merged PR #9211: URL: https://github.com/apache/iceberg/pull/9211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-12-07 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1419477951 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1419451962 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -601,15 +601,76 @@ public void testDropNamespace() { public void testCreateNamespa

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1419451743 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -601,15 +601,76 @@ public void testDropNamespace() { public void testCreateNamespa

[PR] [spark <3.5] [backport from spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
puchengy opened a new pull request, #9243: URL: https://github.com/apache/iceberg/pull/9243 This back ports https://github.com/apache/iceberg/pull/9242. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #9230: URL: https://github.com/apache/iceberg/pull/9230#discussion_r1419439406 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -313,6 +313,37 @@ public void testRecoveryWithoutManifestList() { metadata.currentSnapshot(

Re: [PR] Delta: Fix integration tests and Create DataFile by partition values instead of path [iceberg]

2023-12-07 Thread via GitHub
jackye1995 closed pull request #8398: Delta: Fix integration tests and Create DataFile by partition values instead of path URL: https://github.com/apache/iceberg/pull/8398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Delta: Fix integration tests and Create DataFile by partition values instead of path [iceberg]

2023-12-07 Thread via GitHub
jackye1995 commented on PR #8398: URL: https://github.com/apache/iceberg/pull/8398#issuecomment-1845885416 Most looks good to me, CI seems to fail with unrelated issue, try to retrigger -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-07 Thread via GitHub
jasonf20 commented on code in PR #9230: URL: https://github.com/apache/iceberg/pull/9230#discussion_r1419433713 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -895,7 +895,7 @@ private void cleanUncommittedAppends(Set committed) { }

Re: [PR] Spark 3.5: Rework DeleteFileIndexBenchmark [iceberg]

2023-12-07 Thread via GitHub
flyrain commented on code in PR #9165: URL: https://github.com/apache/iceberg/pull/9165#discussion_r1419397281 ## core/src/test/java/org/apache/iceberg/FileGenerationUtil.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] [spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
puchengy commented on code in PR #9242: URL: https://github.com/apache/iceberg/pull/9242#discussion_r1419414744 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteManifestsProcedure.java: ## @@ -331,4 +331,33 @@ public void testReplacePar

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

2023-12-07 Thread via GitHub
stevenzwu commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1419402421 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expecte

Re: [PR] Core: Create JUnit5 version of TableTestBase [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #9217: URL: https://github.com/apache/iceberg/pull/9217#discussion_r1419398739 ## core/src/test/java/org/apache/iceberg/TestBase.java: ## @@ -0,0 +1,772 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor l

Re: [PR] [spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
RussellSpitzer commented on code in PR #9242: URL: https://github.com/apache/iceberg/pull/9242#discussion_r1419346934 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteManifestsProcedure.java: ## @@ -331,4 +331,33 @@ public void testRepl

Re: [PR] [spark 3.5] Support specifying spec_id in RewriteManifestProcedure [iceberg]

2023-12-07 Thread via GitHub
RussellSpitzer commented on PR #9242: URL: https://github.com/apache/iceberg/pull/9242#issuecomment-1845793619 Changed the title to match the contents -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Api: Track partition statistics via TableMetadata [iceberg]

2023-12-07 Thread via GitHub
ajantha-bhat commented on PR #8502: URL: https://github.com/apache/iceberg/pull/8502#issuecomment-1845791782 @aokolnychyi: Thanks for the review. - I see that most of the questions or comments related to why puffin stats followed that style. We can fix in this PR for partition stats

[PR] [spark 3.5] support specified spec_id in manifest rewrite [iceberg]

2023-12-07 Thread via GitHub
puchengy opened a new pull request, #9242: URL: https://github.com/apache/iceberg/pull/9242 As title. Current manifest rewrite in spark procedure only can be performed on manifests with current spec id. This PR allows user to specify rewrite on manifests with old spec ids. -- This is an

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-07 Thread via GitHub
tmnd1991 commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-1845762590 cc @nastra @dramaticlly @advancedxy for review thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark: IN clause on system function is not pushed down [iceberg]

2023-12-07 Thread via GitHub
tmnd1991 commented on PR #9192: URL: https://github.com/apache/iceberg/pull/9192#issuecomment-1845761731 cc @nastra @dramaticlly @advancedxy for review thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Switch to junit5 for mr [iceberg]

2023-12-07 Thread via GitHub
lschetanrao opened a new pull request, #9241: URL: https://github.com/apache/iceberg/pull/9241 ### Description This PR fixes https://github.com/apache/iceberg/issues/9083 The goal here is to switch all imports to Junit5 and to use AssertJ-style assertions. ### Implementation

Re: [PR] Spark: IN clause on system function is not pushed down [iceberg]

2023-12-07 Thread via GitHub
tmnd1991 commented on code in PR #9192: URL: https://github.com/apache/iceberg/pull/9192#discussion_r1419314263 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala: ## @@ -40,14 +37,20 @@ import org.apache.spark.sql.typ

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-07 Thread via GitHub
tmnd1991 commented on code in PR #9233: URL: https://github.com/apache/iceberg/pull/9233#discussion_r1419309041 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala: ## @@ -39,22 +41,31 @@ import org.apache.spark.sql.typ

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-12-07 Thread via GitHub
cccs-jc commented on PR #8980: URL: https://github.com/apache/iceberg/pull/8980#issuecomment-1845659813 > @cccs-jc i mean let's have changes for 3.5 with it's test only in 3.5 and we can backport the change with it's test in lower spark version like 3.4 and 3.3, 3.4 test failures are expect

Re: [PR] Support parameterized tests at class-level with JUnit5 [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1419205681 ## api/src/test/java/org/apache/iceberg/ParameterizedTestExtension.java: ## @@ -0,0 +1,245 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Core: Optimize manifest evaluation for super wide tables [iceberg]

2023-12-07 Thread via GitHub
irshadcc commented on PR #9147: URL: https://github.com/apache/iceberg/pull/9147#issuecomment-1845560669 > Thanks for raising this @irshadcc, this looks good to me. I've left two small comments, could you take a peek at those? Thanks for fixing this! 🙌 I've added the Javadoc and remov

Re: [I] The "Status" paragraph in the readme seems very outdated [iceberg]

2023-12-07 Thread via GitHub
bitsondatadev commented on issue #9127: URL: https://github.com/apache/iceberg/issues/9127#issuecomment-1845537180 @ronkorving Let me know if you'd like to raise the PR, or I can do it as well. For now pointing to the roadmap page will be the best. Eventually, I'd like there to be something

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419122432 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] maint(transforms): replace `type()` calls with `isinstance()` [iceberg-python]

2023-12-07 Thread via GitHub
Fokko commented on PR #188: URL: https://github.com/apache/iceberg-python/pull/188#issuecomment-1845509371 Thanks for fixing this @jayceslesar 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] maint(transforms): replace `type()` calls with `isinstance()` [iceberg-python]

2023-12-07 Thread via GitHub
Fokko merged PR #188: URL: https://github.com/apache/iceberg-python/pull/188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-07 Thread via GitHub
ConeyLiu commented on code in PR #9233: URL: https://github.com/apache/iceberg/pull/9233#discussion_r1419107868 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala: ## @@ -39,22 +41,31 @@ import org.apache.spark.sql.typ

Re: [I] When will the 0.6.0 version be released? [iceberg-python]

2023-12-07 Thread via GitHub
Fokko commented on issue #192: URL: https://github.com/apache/iceberg-python/issues/192#issuecomment-1845492573 @1taoze The goal is this year, and I'm quite confident that it is realistic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419093010 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [I] Automatically create the tables for the `SqlCatalog` [iceberg-python]

2023-12-07 Thread via GitHub
Fokko closed issue #184: Automatically create the tables for the `SqlCatalog` URL: https://github.com/apache/iceberg-python/issues/184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419095071 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419092132 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419091171 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419090042 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419093588 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Automatically create the tables for the `SqlCatalog` [iceberg-python]

2023-12-07 Thread via GitHub
Fokko merged PR #186: URL: https://github.com/apache/iceberg-python/pull/186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Arrow: Allow missing field-ids from Schema [iceberg-python]

2023-12-07 Thread via GitHub
Fokko commented on code in PR #183: URL: https://github.com/apache/iceberg-python/pull/183#discussion_r1419090834 ## pyiceberg/io/pyarrow.py: ## @@ -713,28 +714,50 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: """Visit a primitive type.""" -def _ge

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-12-07 Thread via GitHub
nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1419087834 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

  1   2   >