Re: [PR] Add clang format [iceberg-cpp]

2024-12-10 Thread via GitHub
zhjwpku commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1877883560 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dis

Re: [PR] feat!: Remove `BoundPartitionSpec` (WIP) [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#discussion_r1877895355 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -273,7 +273,7 @@ mod tests { ))]) .build()?; -let spec = BoundPa

Re: [PR] refactor: avoid async_trait macro for IcebergWriter and provide extra dyn trait for object safety [iceberg-rust]

2024-12-10 Thread via GitHub
ZENOTME commented on PR #760: URL: https://github.com/apache/iceberg-rust/pull/760#issuecomment-2530863619 > Thanks @wenym1 for this pr, could you elaborate the benefit of this change? As you said, this may introduce breaking api change, why we need to do this? One point you mentioned is th

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2024-12-10 Thread via GitHub
felixscherz commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2530867433 I'm also interested, I will have a look at the reference @nicor88 provided and create a PR if I can get something to work:) -- This is an automated message from the Apa

Re: [PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

2024-12-10 Thread via GitHub
Ppei-Wang commented on code in PR #11728: URL: https://github.com/apache/iceberg/pull/11728#discussion_r1877627235 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,24 @@ public String toString() { } } + public static class NullType extends P

Re: [PR] Core: Unimplement Map from CharSequenceMap to obey contract [iceberg]

2024-12-10 Thread via GitHub
findepi commented on code in PR #11704: URL: https://github.com/apache/iceberg/pull/11704#discussion_r1877676085 ## .palantir/revapi.yml: ## @@ -1145,6 +1145,22 @@ acceptedBreaks: new: "method org.apache.iceberg.BaseMetastoreOperations.CommitStatus org.apache.iceberg.Bas

Re: [PR] Add clang format [iceberg-cpp]

2024-12-10 Thread via GitHub
pitrou commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1877780728 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dist

Re: [PR] Add clang format [iceberg-cpp]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1877784467 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distr

Re: [PR] Add clang format [iceberg-cpp]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1877784467 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distr

Re: [PR] Add clang format [iceberg-cpp]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1877784467 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distr

Re: [PR] REST: AuthManager API [iceberg]

2024-12-10 Thread via GitHub
adutra commented on code in PR #10753: URL: https://github.com/apache/iceberg/pull/10753#discussion_r1877564371 ## aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java: ## @@ -204,73 +185,51 @@ private AuthSession authSession() { String token = token

[PR] Core: Fix loading a table in CachingCatalog with metadata table name [iceberg]

2024-12-10 Thread via GitHub
gaborkaszab opened a new pull request, #11738: URL: https://github.com/apache/iceberg/pull/11738 If a regular table had a metadata table name then CachingCatalog throws a NoSuchTableException when loading that table. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Core: Fix loading a table in CachingCatalog with metadata table name [iceberg]

2024-12-10 Thread via GitHub
gaborkaszab commented on PR #11738: URL: https://github.com/apache/iceberg/pull/11738#issuecomment-2531342789 This is an alternative solution to https://github.com/apache/iceberg/pull/11123/ -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Add basic CI to build iceberg and example [iceberg-cpp]

2024-12-10 Thread via GitHub
raulcd commented on code in PR #7: URL: https://github.com/apache/iceberg-cpp/pull/7#discussion_r1877940138 ## .github/workflows/test.yml: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [PR] name mapping serde [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #740: URL: https://github.com/apache/iceberg-rust/pull/740#discussion_r1877955873 ## crates/iceberg/src/spec/mod.rs: ## @@ -20,6 +20,7 @@ mod datatypes; mod manifest; mod manifest_list; +mod name_mapping; Review Comment: I think we can add th

Re: [I] Able to parse name-mapping into a recusive structure. [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko closed issue #723: Able to parse name-mapping into a recusive structure. URL: https://github.com/apache/iceberg-rust/issues/723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] name mapping serde [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #740: URL: https://github.com/apache/iceberg-rust/pull/740#discussion_r1877958397 ## crates/iceberg/src/spec/name_mapping.rs: ## @@ -0,0 +1,307 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

Re: [PR] Add basic CI to build iceberg and example [iceberg-cpp]

2024-12-10 Thread via GitHub
raulcd commented on code in PR #7: URL: https://github.com/apache/iceberg-cpp/pull/7#discussion_r1877919831 ## .github/workflows/test.yml: ## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [PR] refactor: avoid async_trait for FileWrite and provide extra dyn methods [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #762: URL: https://github.com/apache/iceberg-rust/pull/762#discussion_r1877920195 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -213,7 +213,7 @@ impl SchemaVisitor for IndexByParquetPathName { pub struct ParquetWriter { sch

Re: [PR] Add basic CI to build iceberg and example [iceberg-cpp]

2024-12-10 Thread via GitHub
raulcd commented on code in PR #7: URL: https://github.com/apache/iceberg-cpp/pull/7#discussion_r1877920379 ## .github/workflows/test.yml: ## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [PR] Add basic CI to build iceberg and example [iceberg-cpp]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #7: URL: https://github.com/apache/iceberg-cpp/pull/7#discussion_r1877929785 ## .github/workflows/test.yml: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [PR] Add basic CI to build iceberg and example [iceberg-cpp]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #7: URL: https://github.com/apache/iceberg-cpp/pull/7#discussion_r1877928847 ## .github/workflows/test.yml: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

2024-12-10 Thread via GitHub
Ppei-Wang commented on code in PR #11728: URL: https://github.com/apache/iceberg/pull/11728#discussion_r1877637942 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,24 @@ public String toString() { } } + public static class NullType extends P

Re: [PR] feat!: Remove `BoundPartitionSpec` (WIP) [iceberg-rust]

2024-12-10 Thread via GitHub
c-thiel commented on code in PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#discussion_r1877599173 ## crates/iceberg/src/spec/partition.rs: ## @@ -244,21 +173,21 @@ pub struct UnboundPartitionField { /// Unbound partition spec can be built without a schema and la

Re: [PR] Core: Fix caching table with metadata table names [iceberg]

2024-12-10 Thread via GitHub
gaborkaszab commented on code in PR #11123: URL: https://github.com/apache/iceberg/pull/11123#discussion_r1877738556 ## core/src/main/java/org/apache/iceberg/CachingCatalog.java: ## @@ -145,22 +146,26 @@ public Table loadTable(TableIdentifier ident) { } if (MetadataT

[PR] Spark 3.5: Add ignore-invalid-options to RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction [iceberg]

2024-12-10 Thread via GitHub
manuzhang opened a new pull request, #11737: URL: https://github.com/apache/iceberg/pull/11737 This PR adds a new option `ignore-invalid-options` to RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction. It can be useful when we canary-release a new Iceberg version wi

Re: [PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

2024-12-10 Thread via GitHub
Ppei-Wang commented on code in PR #11728: URL: https://github.com/apache/iceberg/pull/11728#discussion_r1877627235 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,24 @@ public String toString() { } } + public static class NullType extends P

Re: [PR] name mapping serde [iceberg-rust]

2024-12-10 Thread via GitHub
Fokko merged PR #740: URL: https://github.com/apache/iceberg-rust/pull/740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Fix caching table with metadata table names [iceberg]

2024-12-10 Thread via GitHub
gaborkaszab commented on code in PR #11123: URL: https://github.com/apache/iceberg/pull/11123#discussion_r1878043833 ## core/src/main/java/org/apache/iceberg/CachingCatalog.java: ## @@ -145,22 +146,26 @@ public Table loadTable(TableIdentifier ident) { } if (MetadataT

[PR] AWS: adding s3 tables catalog [iceberg]

2024-12-10 Thread via GitHub
stubz151 opened a new pull request, #11739: URL: https://github.com/apache/iceberg/pull/11739 ## Why am I doing this The catalog in a separate github package isn't a long term solution. This is a better integration path for iceberg customers. ## What Am I doing - Adding the [s3

Re: [PR] Update StrictProjection tests [iceberg-python]

2024-12-10 Thread via GitHub
Fokko commented on code in PR #1422: URL: https://github.com/apache/iceberg-python/pull/1422#discussion_r1879026523 ## pyiceberg/transforms.py: ## @@ -985,7 +985,7 @@ def _truncate_number_strict( elif isinstance(pred, BoundGreaterThanOrEqual): return GreaterThan(Re

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-12-10 Thread via GitHub
amogh-jahagirdar commented on PR #11420: URL: https://github.com/apache/iceberg/pull/11420#issuecomment-2533397879 I'll go ahead and merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Core, Flink, Spark, KafkaConnect: Remove remaining usage of deprecated path API [iceberg]

2024-12-10 Thread via GitHub
amogh-jahagirdar commented on code in PR #11744: URL: https://github.com/apache/iceberg/pull/11744#discussion_r1879171908 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestExpireSnapshotsProcedure.java: ## @@ -300,7 +300,7 @@ public void testEx

Re: [PR] Core, Flink, Spark, KafkaConnect: Remove remaining usage of deprecated path API [iceberg]

2024-12-10 Thread via GitHub
amogh-jahagirdar commented on code in PR #11744: URL: https://github.com/apache/iceberg/pull/11744#discussion_r1879171773 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java: ## @@ -148,7 +148,7 @@ public boolean next() throws IOException { i

Re: [PR] Spec: Support geo type [iceberg]

2024-12-10 Thread via GitHub
szehon-ho commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1879110520 ## format/spec.md: ## @@ -1102,6 +1105,7 @@ Hash results are not dependent on decimal scale, which is part of the type, not 4. UUIDs are encoded using big endian.

Re: [PR] Core: Fix failure when reading files table with branch [iceberg]

2024-12-10 Thread via GitHub
szehon-ho commented on PR #11719: URL: https://github.com/apache/iceberg/pull/11719#issuecomment-2533425414 Is it ready for review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Core: Fix failure when reading files table with branch [iceberg]

2024-12-10 Thread via GitHub
szehon-ho commented on PR #11719: URL: https://github.com/apache/iceberg/pull/11719#issuecomment-2533424881 This makes sense to me. How about other metadata tables? @dramaticlly do you also want take a look? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-12-10 Thread via GitHub
Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2533696010 > Thanks @Guosmilesmile for the PR, and @mxm and @stevenzwu for the review! > > @Guosmilesmile please prepare the backport commits to the other Flink versions. > > Th

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

2024-12-10 Thread via GitHub
liurenjie1024 commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2533708338 Hi, @feniljain I also didn't find how distinct counts are implemented in java, but according to the spec it's supposed to be an estimated value using sketch. I think we coul

Re: [PR] Flink: Backport #11662 Fix range distribution npe when value is null to Flink 1.18 and 1.19 [iceberg]

2024-12-10 Thread via GitHub
Guosmilesmile commented on PR #11745: URL: https://github.com/apache/iceberg/pull/11745#issuecomment-2533693064 @pvary here's the PR to backport for Fix range distribution npe when value is null to Flink v1.18 and v1.19. Could you please take some time to help review it? -- This

Re: [PR] Flink: Backport #11662 Fix range distribution npe when value is null to Flink 1.18 and 1.19 [iceberg]

2024-12-10 Thread via GitHub
pvary commented on PR #11745: URL: https://github.com/apache/iceberg/pull/11745#issuecomment-2533701394 @Guosmilesmile: Was this a clean backport, or you had to modify things to apply the changes? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

2024-12-10 Thread via GitHub
feniljain commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2533731058 >but according to the spec it's supposed to be an estimated value using sketch That sounds interesting, thanks for the link up to spec! >I think we could start with

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-12-10 Thread via GitHub
rahil-c commented on PR #11369: URL: https://github.com/apache/iceberg/pull/11369#issuecomment-2532787657 @amogh-jahagirdar Was wondering if there is another issues, or if you think we can land this? -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] docker: The `archive` seems unstable [iceberg-rust]

2024-12-10 Thread via GitHub
liurenjie1024 merged PR #773: URL: https://github.com/apache/iceberg-rust/pull/773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] docker: The `archive` seems unstable [iceberg-rust]

2024-12-10 Thread via GitHub
liurenjie1024 commented on PR #773: URL: https://github.com/apache/iceberg-rust/pull/773#issuecomment-2533460061 Thanks @Fokko for fixing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] refactor: avoid async_trait for FileWrite and provide extra dyn methods [iceberg-rust]

2024-12-10 Thread via GitHub
wenym1 commented on code in PR #762: URL: https://github.com/apache/iceberg-rust/pull/762#discussion_r1879426201 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -213,7 +213,7 @@ impl SchemaVisitor for IndexByParquetPathName { pub struct ParquetWriter { sc

[I] Discussion: make DataFile Serializable && Deserializable [iceberg-rust]

2024-12-10 Thread via GitHub
ZENOTME opened a new issue, #774: URL: https://github.com/apache/iceberg-rust/issues/774 ## Context Make Datafile Serializable && Deserializable is useful, e.g. In distributed compute engine, it will create multiple writers in multiple machines and write the data in parallel and get

<    1   2