Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-02 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2031222529 Thanks for the review @marvinlanhenke ! I'll address the comments today or tomorrow morning 😁 -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-02 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2031227629 Also I've got some uncommitted changes around having a boolean evaluator trait, similar to your suggestion. It has default implementations for and / or / not / always true / always false,

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on code in PR #10069: URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on code in PR #10069: URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata

Re: [PR] Build: add gradle wrapper validation action [iceberg]

2024-04-02 Thread via GitHub
nastra commented on PR #10073: URL: https://github.com/apache/iceberg/pull/10073#issuecomment-2031249365 thanks for opening the PR, but I don't think it applies for Iceberg, because we don't have the jar committed as can be seen in https://github.com/apache/iceberg/tree/main/gradle/wrapper.

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-02 Thread via GitHub
nastra commented on PR #10067: URL: https://github.com/apache/iceberg/pull/10067#issuecomment-2031250673 this is most likely due to https://github.com/apache/iceberg/issues/10060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on code in PR #10069: URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on PR #10069: URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2031250719 @lurnagao-dahua please check styles. > The reason is that in some cases, the e.getMessage() return null and it will throw NullPointerException, then skip checkCommitStatus, it

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10067: URL: https://github.com/apache/iceberg/pull/10067#discussion_r1547281452 ## site/docs/blogs.md: ## @@ -23,6 +23,37 @@ title: "Blogs" Here is a list of company blogs that talk about Iceberg. The blogs are ordered from most recent to oldest

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547287086 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -448,6 +446,47 @@ static HttpRequestInterceptor loadInterceptorDynamically( return instance;

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547287402 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -448,6 +446,47 @@ static HttpRequestInterceptor loadInterceptorDynamically( return instance;

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547290305 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -448,6 +446,47 @@ static HttpRequestInterceptor loadInterceptorDynamically( return instance;

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547292782 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547292782 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547299122 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547300740 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547301269 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547301639 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547304325 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547305188 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1547300740 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -133,6 +136,59 @@ public void testDynamicHttpRequestInterceptorLoading() { assertThat(((T

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
ajantha-bhat commented on PR #9870: URL: https://github.com/apache/iceberg/pull/9870#issuecomment-2031310056 cc: @jbonofre -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
Fokko commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1547395200 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An optional pr

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1547402202 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +232,25 @@ public void testMigrateEmptyTa

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1547402202 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +232,25 @@ public void testMigrateEmptyTa

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
ajantha-bhat commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1547408417 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An opti

Re: [I] Disable checking links for Blogs section [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on issue #10060: URL: https://github.com/apache/iceberg/issues/10060#issuecomment-2031439818 I also see errors when checking maven repo. ``` ERROR: 21 dead links found! [✖] https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runti

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1547447584 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +232,25 @@ public void testMigrateEmpt

Re: [PR] Support identifier warehouses [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on PR #308: URL: https://github.com/apache/iceberg-rust/pull/308#issuecomment-2031513654 > Hi, @Fokko Thanks for this fix. It also reminds me that should we append the warehouse parameter to `getConfig` call? Seems we already have that. -- This is an automa

Re: [PR] Support identifier warehouses [iceberg-rust]

2024-04-02 Thread via GitHub
Fokko commented on code in PR #308: URL: https://github.com/apache/iceberg-rust/pull/308#discussion_r1547503192 ## crates/catalog/rest/src/catalog.rs: ## @@ -617,7 +617,13 @@ impl RestCatalog { props.extend(config); } -let file_io = match self.co

Re: [PR] Support identifier warehouses [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #308: URL: https://github.com/apache/iceberg-rust/pull/308#discussion_r1547526634 ## crates/catalog/rest/src/catalog.rs: ## @@ -617,7 +617,13 @@ impl RestCatalog { props.extend(config); } -let file_io = match

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1547529529 ## data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java: ## @@ -171,6 +176,10 @@ public static List listPartition( } } + public static boolean

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1547543252 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -84,16 +86,16 @@ private static void assertEquals(Type type, Object expected, Object actual) {

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547488931 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a giv

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-02 Thread via GitHub
manuzhang commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1547578684 ## data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java: ## @@ -171,6 +176,10 @@ public static List listPartition( } } + public static bool

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1547582654 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#issuecomment-2031700958 > Hi, @marvinlanhenke Thanks for pr, it looks great! I have some small suggestion to restructure the code to make it easier for review. Really greatful for these tests! T

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547632925 ## data/src/test/java/org/apache/iceberg/io/TestRollingFileWriters.java: ## @@ -20,60 +20,60 @@ import java.io.File; import java.io.IOException; +import java.nio.f

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547631706 ## data/src/test/java/org/apache/iceberg/io/TestPositionDeltaWriters.java: ## @@ -20,43 +20,42 @@ import java.io.File; import java.io.IOException; +import java.nio

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547634234 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestDeltaTaskWriter.java: ## @@ -65,31 +68,28 @@ import org.apache.iceberg.util.StructLikeSet; impor

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547636721 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergFilesCommitter.java: ## @@ -73,44 +77,39 @@ import org.apache.iceberg.util.ThreadPools; i

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
tomtongue commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547639555 ## data/src/test/java/org/apache/iceberg/io/TestPositionDeltaWriters.java: ## @@ -20,43 +20,42 @@ import java.io.File; import java.io.IOException; +import java.

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547639860 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a gi

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547646211 ## data/src/test/java/org/apache/iceberg/io/TestPositionDeltaWriters.java: ## @@ -20,43 +20,42 @@ import java.io.File; import java.io.IOException; +import java.nio

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
tomtongue commented on code in PR #10063: URL: https://github.com/apache/iceberg/pull/10063#discussion_r1547650695 ## data/src/test/java/org/apache/iceberg/io/TestPositionDeltaWriters.java: ## @@ -20,43 +20,42 @@ import java.io.File; import java.io.IOException; +import java.

Re: [PR] Migrate TableTestBase related classes to JUnit5 and delete TableTestBase [iceberg]

2024-04-02 Thread via GitHub
nastra merged PR #10063: URL: https://github.com/apache/iceberg/pull/10063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547653811 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a gi

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547649993 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a giv

Re: [PR] Spark 3.3: drop_namespace with CASCADE support [iceberg]

2024-04-02 Thread via GitHub
supsupsap commented on PR #7275: URL: https://github.com/apache/iceberg/pull/7275#issuecomment-2031741342 @abmo-x do you plan to merge this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547667177 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a gi

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547676828 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a giv

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547681855 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a gi

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547785953 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a gi

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2031946776 > @liurenjie1024 @ZENOTME @sdd @Xuanwo I'd really appreciate your thoughts on this: > > I took a closer look at the work @sdd has already done - and I think in order to pr

Re: [PR] feat: Project transform [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1547819858 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,300 @@ impl Transform { _ => self == other, } } + +/// Projects a giv

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-02 Thread via GitHub
Fokko commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1547810906 ## pyiceberg/manifest.py: ## @@ -289,10 +286,7 @@ def partition_field_to_data_file_partition_field(partition_field_type: IcebergTy @partition_field_to_data_fil

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
ajantha-bhat commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1547852220 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An opti

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
jbonofre commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1547920214 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An optional

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-04-02 Thread via GitHub
nastra commented on PR #9940: URL: https://github.com/apache/iceberg/pull/9940#issuecomment-2032092212 > I've got strong concerns about using `enum` here - special handling here and there, I think, that complicates things for adopters of any OpenAPI spec. @snazy we use `enum` in t

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
lurnagao-dahua commented on PR #10069: URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2032181736 > @lurnagao-dahua请检查样式。 > > > 原因是在某些情况下,e.getMessage()返回null,会抛出NullPointerException,然后跳过checkCommitStatus,可能是删除metadataLocation,实际上元数据提交成功。 > > 是否可以为这种情况添加 UT?

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
lurnagao-dahua closed pull request #10069: Hive: Fix metadata file not found URL: https://github.com/apache/iceberg/pull/10069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-02 Thread via GitHub
lurnagao-dahua commented on PR #10069: URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2032193393 > @lurnagao-dahua please check styles. > > > The reason is that in some cases, the e.getMessage() return null and it will throw NullPointerException, then skip checkCommitS

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-04-02 Thread via GitHub
snazy commented on PR #9940: URL: https://github.com/apache/iceberg/pull/9940#issuecomment-2032202321 > > I've got strong concerns about using `enum` here - special handling here and there, I think, that complicates things for adopters of any OpenAPI spec. > > @snazy we use `enum` in

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-02 Thread via GitHub
marvinlanhenke commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2032233828 > which is somehow motivated by @viirya 's [pr](https://github.com/apache/iceberg-rust/pull/295/files#diff-a59622727cd67153abdf02031475bf8a1b1921738df4ca9903a685ff6970b7aaR472),

[PR] Add option to delete datafiles [iceberg-python]

2024-04-02 Thread via GitHub
Fokko opened a new pull request, #569: URL: https://github.com/apache/iceberg-python/pull/569 This is done through the Iceberg metadata, resulting in efficient deletes if the data is partitioned correctly -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-02 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1548068306 ## pyiceberg/typedef.py: ## @@ -199,3 +199,7 @@ def __repr__(self) -> str: def record_fields(self) -> List[str]: """Return values of all the fields of

[I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-02 Thread via GitHub
AwasthiSomesh opened a new issue, #10078: URL: https://github.com/apache/iceberg/issues/10078 Hi Team , We are doing below code to access iceberg table from glue catalog and data storage as S3 var spark = SparkSession.builder().master("local[*]") .config("spark.sql.def

Re: [PR] Improve CLI Text by Adding Verbose Text for Commands [iceberg-go]

2024-04-02 Thread via GitHub
zeroshade commented on code in PR #68: URL: https://github.com/apache/iceberg-go/pull/68#discussion_r1548127844 ## cmd/iceberg/main.go: ## @@ -34,16 +34,21 @@ import ( const usage = `iceberg. Usage: - iceberg list [options] [PARENT] - iceberg describe [options] [namespace

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-02 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1548140319 ## pyiceberg/table/__init__.py: ## @@ -2526,25 +2537,44 @@ def _dataframe_to_data_files( """ from pyiceberg.io.pyarrow import bin_pack_arrow_table, write_

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-02 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1548140319 ## pyiceberg/table/__init__.py: ## @@ -2526,25 +2537,44 @@ def _dataframe_to_data_files( """ from pyiceberg.io.pyarrow import bin_pack_arrow_table, write_

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-02 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1548140319 ## pyiceberg/table/__init__.py: ## @@ -2526,25 +2537,44 @@ def _dataframe_to_data_files( """ from pyiceberg.io.pyarrow import bin_pack_arrow_table, write_

Re: [I] Unable to load an iceberg table from aws glue catalog [iceberg-python]

2024-04-02 Thread via GitHub
geruh commented on issue #515: URL: https://github.com/apache/iceberg-python/issues/515#issuecomment-2032913100 No Problem!! This could potentially be a bug if we assume that the catalog and FileIO (S3) share the same aws profile configs. On one side, having a single profile configu

Re: [I] Snowflake Iceberg Partitioned data read issue [iceberg]

2024-04-02 Thread via GitHub
findinpath commented on issue #9404: URL: https://github.com/apache/iceberg/issues/9404#issuecomment-2032964716 @sfc-gh-rortloff i went through the Snowflake documenation https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table and don't see any reference related to partitioning

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-02 Thread via GitHub
rahil-c commented on PR #9695: URL: https://github.com/apache/iceberg/pull/9695#issuecomment-2033006567 @nastra @rdblue @danielcweeks @jackye1995 @amogh-jahagirdar When looking again at the `capabalities` pr: https://github.com/apache/iceberg/pull/9940, are we sure we want to add sc

[I] PyArrow S3FileSystem doesn't honor the AWS profile config [iceberg-python]

2024-04-02 Thread via GitHub
geruh opened a new issue, #570: URL: https://github.com/apache/iceberg-python/issues/570 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 When initializing the GlueCatalog with a specific AWS profile, everything works as it should with catalog

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-02 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1548617271 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/comet/CometIcebergColumnReader.java: ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apac

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-02 Thread via GitHub
AlexMercedCoder commented on code in PR #10067: URL: https://github.com/apache/iceberg/pull/10067#discussion_r1548618808 ## site/docs/blogs.md: ## @@ -23,6 +23,37 @@ title: "Blogs" Here is a list of company blogs that talk about Iceberg. The blogs are ordered from most recent

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-02 Thread via GitHub
AlexMercedCoder commented on code in PR #10067: URL: https://github.com/apache/iceberg/pull/10067#discussion_r1548619662 ## site/docs/blogs.md: ## @@ -23,6 +23,37 @@ title: "Blogs" Here is a list of company blogs that talk about Iceberg. The blogs are ordered from most recent

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-02 Thread via GitHub
huaxingao commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1548708375 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/comet/CometIcebergColumnReader.java: ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache So

Re: [I] Cannot write incompatible dataset to table with schema error for list types [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2192: URL: https://github.com/apache/iceberg/issues/2192#issuecomment-2033310543 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Cannot write incompatible dataset to table with schema error for list types [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] closed issue #2192: Cannot write incompatible dataset to table with schema error for list types URL: https://github.com/apache/iceberg/issues/2192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Hive: got error while joining iceberg table and hive table [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] closed issue #2198: Hive: got error while joining iceberg table and hive table URL: https://github.com/apache/iceberg/issues/2198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Hive: got error while joining iceberg table and hive table [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2198: URL: https://github.com/apache/iceberg/issues/2198#issuecomment-2033310562 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Need help inserting data into hadoop table with flink sql in java [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] closed issue #2209: Need help inserting data into hadoop table with flink sql in java URL: https://github.com/apache/iceberg/issues/2209 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Need help inserting data into hadoop table with flink sql in java [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2209: URL: https://github.com/apache/iceberg/issues/2209#issuecomment-2033310589 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] how to fix org.apache.spark.shuffle.FetchFailedException: [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] closed issue #2211: how to fix org.apache.spark.shuffle.FetchFailedException: URL: https://github.com/apache/iceberg/issues/2211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] how to fix org.apache.spark.shuffle.FetchFailedException: [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2211: URL: https://github.com/apache/iceberg/issues/2211#issuecomment-2033310616 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink cdc events with update or delete doesn't work in 0.11.0 branch [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2409: URL: https://github.com/apache/iceberg/issues/2409#issuecomment-2033310842 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Please remove old releases [iceberg]

2024-04-02 Thread via GitHub
github-actions[bot] commented on issue #2414: URL: https://github.com/apache/iceberg/issues/2414#issuecomment-2033310864 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-02 Thread via GitHub
syun64 commented on PR #498: URL: https://github.com/apache/iceberg-python/pull/498#issuecomment-2033344103 > Shall we move "append", "overwrite", and "add_files" to `Transaction` class? This change would enable us to seamlessly chain these operations with other table updates in a single co

Re: [PR] feat: support uri redirect in rest client [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 merged PR #310: URL: https://github.com/apache/iceberg-rust/pull/310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-02 Thread via GitHub
liurenjie1024 commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2033371130 > > which is somehow motivated by @viirya 's [pr](https://github.com/apache/iceberg-rust/pull/295/files#diff-a59622727cd67153abdf02031475bf8a1b1921738df4ca9903a685ff6970b7aaR472),

[PR] refine: seperate parquet reader and arrow convert [iceberg-rust]

2024-04-02 Thread via GitHub
ZENOTME opened a new pull request, #313: URL: https://github.com/apache/iceberg-rust/pull/313 This PR separates out the parquet reader from the arrow module. And make the arrow module a dir so that we can separate the `from_arow` and `to_arrow`. -- This is an automated message from the A

Re: [PR] refine: seperate parquet reader and arrow convert [iceberg-rust]

2024-04-02 Thread via GitHub
ZENOTME commented on PR #313: URL: https://github.com/apache/iceberg-rust/pull/313#issuecomment-2033373684 cc @liurenjie1024 @Xuanwo @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-02 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1548810072 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-02 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1548809949 ## crates/iceberg/src/arrow.rs: ## @@ -20,24 +20,38 @@ use async_stream::try_stream; use futures::stream::StreamExt; use parquet::arrow::{ParquetRecordBatchStreamB

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-02 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1548813765 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

[PR] Move writes to Transaction [iceberg-python]

2024-04-02 Thread via GitHub
syun64 opened a new pull request, #571: URL: https://github.com/apache/iceberg-python/pull/571 As a followup from @HonahX 's suggestion on https://github.com/apache/iceberg-python/pull/498 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-02 Thread via GitHub
ajantha-bhat commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1548815679 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An opti

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-02 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1548817290 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

  1   2   >