Re: [PR] Update the docs for working with Flink and REST catalog [iceberg]

2025-04-14 Thread via GitHub
dttung2905 commented on code in PR #12726: URL: https://github.com/apache/iceberg/pull/12726#discussion_r2042892815 ## docs/docs/flink-connector.md: ## @@ -105,7 +126,6 @@ CREATE TABLE flink_table ( ... ); ``` - Review Comment: Sorry my bad. Fixed it -- This is a

Re: [PR] Refactor `Metadata` in `Transaction` [iceberg-python]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #1903: URL: https://github.com/apache/iceberg-python/pull/1903#discussion_r2042815333 ## tests/integration/test_writes/test_writes.py: ## @@ -1683,3 +1683,33 @@ def test_write_optional_list(session_catalog: Catalog) -> None: session_catalog.lo

Re: [PR] Update the docs for working with Flink and REST catalog [iceberg]

2025-04-14 Thread via GitHub
dttung2905 commented on code in PR #12726: URL: https://github.com/apache/iceberg/pull/12726#discussion_r2042892201 ## docs/docs/flink.md: ## @@ -215,13 +215,17 @@ CREATE CATALOG hive_catalog WITH ( ); ``` -The following properties can be set if using the Hive catalog: Revi

Re: [PR] feat: snapshot serde [iceberg-cpp]

2025-04-14 Thread via GitHub
wgtmac commented on code in PR #74: URL: https://github.com/apache/iceberg-cpp/pull/74#discussion_r2043357992 ## src/iceberg/json_internal.cc: ## @@ -231,6 +296,53 @@ nlohmann::json SchemaToJson(const Schema& schema) { return json; } +nlohmann::json SnapshotRefToJson(const

Re: [PR] feat: snapshot serde [iceberg-cpp]

2025-04-14 Thread via GitHub
wgtmac commented on code in PR #74: URL: https://github.com/apache/iceberg-cpp/pull/74#discussion_r2043358743 ## src/iceberg/json_internal.cc: ## @@ -231,6 +296,53 @@ nlohmann::json SchemaToJson(const Schema& schema) { return json; } +nlohmann::json SnapshotRefToJson(const

Re: [PR] Spark 3.5: Add Parallelism Parameter Validation to AddFilesProcedure. [iceberg]

2025-04-14 Thread via GitHub
slfan1989 commented on PR #12784: URL: https://github.com/apache/iceberg/pull/12784#issuecomment-2803523254 @manuzhang @ebyhr Could you help review this PR? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] fix(catalog/rest): Fix concurrency bug in REST catalog request signing [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on PR #384: URL: https://github.com/apache/iceberg-go/pull/384#issuecomment-2803065824 You should be able to inject credentials with the `WithAwsConfig` option for constructing the catalog, right? -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-14 Thread via GitHub
jbonofre commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2042840223 ## format/spec.md: ## @@ -1605,13 +1611,8 @@ All readers are required to read tables with unknown partition transforms, ignor Writing v3 metadata: * Partition F

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2042855103 ## format/spec.md: ## @@ -1605,13 +1611,8 @@ All readers are required to read tables with unknown partition transforms, ignor Writing v3 metadata: * Partition Fiel

Re: [PR] Flink: Move unlock from MemoryLock open to TestCase Before [iceberg]

2025-04-14 Thread via GitHub
Guosmilesmile commented on code in PR #12793: URL: https://github.com/apache/iceberg/pull/12793#discussion_r2042873689 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java: ## @@ -62,6 +62,8 @@ void before() { Table table

Re: [I] How to query table by partition filed? [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #11329: URL: https://github.com/apache/iceberg/issues/11329#issuecomment-2803390971 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] feat: snapshot serde [iceberg-cpp]

2025-04-14 Thread via GitHub
lidavidm commented on code in PR #74: URL: https://github.com/apache/iceberg-cpp/pull/74#discussion_r2043228407 ## test/json_internal_test.cc: ## @@ -148,4 +148,96 @@ TEST(JsonPartitionTest, PartitionSpec) { EXPECT_EQ(spec, *parsed_spec_result.value()); } +TEST(JsonInterna

Re: [PR] feat: snapshot serde [iceberg-cpp]

2025-04-14 Thread via GitHub
wgtmac commented on code in PR #74: URL: https://github.com/apache/iceberg-cpp/pull/74#discussion_r2042433243 ## src/iceberg/snapshot.h: ## @@ -54,13 +77,35 @@ struct ICEBERG_EXPORT SnapshotRef { /// of the snapshot reference to keep while expiring snapshots. Defaults to t

Re: [PR] feat: support strict projection [iceberg-rust]

2025-04-14 Thread via GitHub
liurenjie1024 merged PR #946: URL: https://github.com/apache/iceberg-rust/pull/946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spec: Update row lineage requirements for upgrading tables [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12781: URL: https://github.com/apache/iceberg/pull/12781#discussion_r2043112788 ## format/spec.md: ## @@ -736,8 +738,6 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on The snapshot's `first-row-id` is the star

Re: [I] Patch release request CVE-2025-30065 [iceberg]

2025-04-14 Thread via GitHub
nastra commented on issue #12749: URL: https://github.com/apache/iceberg/issues/12749#issuecomment-2803984804 In the past the community has always recommended users to upgrade to the latest version. The reason I thought it might be a good idea to get this into 1.8.x (and not any older versi

Re: [I] Track: fix minimal version of dependencies [iceberg-rust]

2025-04-14 Thread via GitHub
liurenjie1024 commented on issue #1000: URL: https://github.com/apache/iceberg-rust/issues/1000#issuecomment-2803540594 I think we could close this issue for now? https://github.com/apache/iceberg-rust/pull/1209 has added minimum deps in our `Cargo.toml`. -- This is an automated message

Re: [PR] Spark: prefix SparkTable with 'iceberg' to clearly identify Iceberg table [iceberg]

2025-04-14 Thread via GitHub
cgpoh commented on PR #12543: URL: https://github.com/apache/iceberg/pull/12543#issuecomment-2803679796 > @wypoon , the motivation for this PR is when I’m trying to capture data lineage using DataHub in spark streaming mode. In the DataHub [code](https://github.com/datahub-project/datahub/b

Re: [PR] feat: add json serde for table metadata [iceberg-cpp]

2025-04-14 Thread via GitHub
wgtmac commented on PR #75: URL: https://github.com/apache/iceberg-cpp/pull/75#issuecomment-2803730355 @lidavidm @zhjwpku @gty404 Could you help review this? I will add test cases once https://github.com/apache/iceberg-cpp/pull/74 has been merged. Note that I have removed `Form

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
nastra commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042564996 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -2272,7 +2280,7 @@ public void testCompleteCreateOrReplaceTransactionCreate() { if (!ov

Re: [PR] Build: Bump nessie from 0.103.2 to 0.103.3 [iceberg]

2025-04-14 Thread via GitHub
nastra merged PR #12786: URL: https://github.com/apache/iceberg/pull/12786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Patch release request CVE-2025-30065 [iceberg]

2025-04-14 Thread via GitHub
jeremysong commented on issue #12749: URL: https://github.com/apache/iceberg/issues/12749#issuecomment-2803680118 > If this is the main concern, then I think our recommendation should be to update the version of the Iceberg runtime bundle. @rdblue But the newer version of the Iceberg

Re: [PR] Core: Support first-row-id for manifests and manifest lists [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12672: URL: https://github.com/apache/iceberg/pull/12672#discussion_r2043087323 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -283,11 +285,26 @@ public Snapshot apply() { throw new RuntimeIOException(e, "Failed to wr

Re: [I] Variant Data Type Support [iceberg]

2025-04-14 Thread via GitHub
mailmahee commented on issue #10392: URL: https://github.com/apache/iceberg/issues/10392#issuecomment-2803422395 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[PR] Retry on NoSuchNamespaceException not found in rename table for rest catalog [iceberg]

2025-04-14 Thread via GitHub
huan233usc opened a new pull request, #12159: URL: https://github.com/apache/iceberg/pull/12159 This PR attempts to fix the issue mentioned in https://github.com/apache/iceberg/issues/11154. On NoSuchNamespaceException, we retry rename table with stripping the catalog name if the fir

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
talatuyarer commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042637091 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -2272,7 +2280,7 @@ public void testCompleteCreateOrReplaceTransactionCreate() { if

Re: [PR] Add table property to disable/enable parquet column statistics #12770 [iceberg]

2025-04-14 Thread via GitHub
huaxiangsun commented on code in PR #12771: URL: https://github.com/apache/iceberg/pull/12771#discussion_r2043338512 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetWriter.java: ## @@ -151,4 +152,27 @@ public void testFpp() throws IOException, No

Re: [PR] feat: add json serde for table metadata [iceberg-cpp]

2025-04-14 Thread via GitHub
gty404 commented on code in PR #75: URL: https://github.com/apache/iceberg-cpp/pull/75#discussion_r2043545911 ## src/iceberg/statistics_file.h: ## @@ -99,6 +93,29 @@ struct ICEBERG_EXPORT PartitionStatisticsFile { std::string path; /// The size of the partition statistics

Re: [I] How to use iceberg metadata or something to get which partition of table does spark sql hit? [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #11331: URL: https://github.com/apache/iceberg/issues/11331#issuecomment-2803391012 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] RewriteManifest with more options [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #9615: URL: https://github.com/apache/iceberg/issues/9615#issuecomment-2803390415 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] org.apache.iceberg.parquet.PruneColumns does not project properly for depth 3 structs. [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #11332: URL: https://github.com/apache/iceberg/issues/11332#issuecomment-2803391046 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Unable to query iceberg table , getting unable to open manifest file "org.apache.avro.InvalidAvroMagicException: Not an Avro data file" [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] closed issue #11070: Unable to query iceberg table , getting unable to open manifest file "org.apache.avro.InvalidAvroMagicException: Not an Avro data file" URL: https://github.com/apache/iceberg/issues/11070 -- This is an automated message from the Apache Git Service. To

Re: [I] Can't select table If drop the corresponding column after replacing or dropping partition spec field [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #11314: URL: https://github.com/apache/iceberg/issues/11314#issuecomment-2803390931 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Unable to query iceberg table , getting unable to open manifest file "org.apache.avro.InvalidAvroMagicException: Not an Avro data file" [iceberg]

2025-04-14 Thread via GitHub
github-actions[bot] commented on issue #11070: URL: https://github.com/apache/iceberg/issues/11070#issuecomment-2803390795 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Fix for metadata entries table for MOR tables containing Delete Files. [iceberg-python]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #1902: URL: https://github.com/apache/iceberg-python/pull/1902#discussion_r2042825738 ## pyiceberg/table/inspect.py: ## @@ -205,9 +205,13 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: "rec

Re: [PR] refactor(table): move all action names to constants [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on code in PR #392: URL: https://github.com/apache/iceberg-go/pull/392#discussion_r2042962462 ## table/updates.go: ## @@ -25,15 +25,26 @@ import ( ) const ( - updateSpec = "add-spec" - updateAddSchema= "add-schema" -

Re: [PR] feat: add json serde for table metadata [iceberg-cpp]

2025-04-14 Thread via GitHub
wgtmac commented on code in PR #75: URL: https://github.com/apache/iceberg-cpp/pull/75#discussion_r2043558155 ## src/iceberg/statistics_file.h: ## @@ -99,6 +93,29 @@ struct ICEBERG_EXPORT PartitionStatisticsFile { std::string path; /// The size of the partition statistics

Re: [PR] Spec: Make next-row-id required in v3 [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on PR #12757: URL: https://github.com/apache/iceberg/pull/12757#issuecomment-2803358324 After working on the implementation and posting a PR to update the spec for how we handle upgrades, I think that this PR is correct and that `next-row-id` can be required in v3. Rather t

Re: [PR] Spec: Update row lineage requirements for upgrading tables [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12781: URL: https://github.com/apache/iceberg/pull/12781#discussion_r2043127255 ## format/spec.md: ## @@ -786,9 +786,9 @@ Notes: First Row ID Assignment -When adding a new data manifest file, its `first_row_id` field is assigned the val

Re: [PR] Spec: Update row lineage requirements for upgrading tables [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12781: URL: https://github.com/apache/iceberg/pull/12781#discussion_r2043132199 ## format/spec.md: ## @@ -786,9 +786,9 @@ Notes: First Row ID Assignment -When adding a new data manifest file, its `first_row_id` field is assigned the val

Re: [PR] Introduce AuthManager [iceberg-python]

2025-04-14 Thread via GitHub
sungwy merged PR #1908: URL: https://github.com/apache/iceberg-python/pull/1908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
kumarpritam863 commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042421471 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,32 @@ */ package org.apache.iceberg.connect;

Re: [PR] Spark 4.0 integration [iceberg]

2025-04-14 Thread via GitHub
huaxingao commented on PR #12494: URL: https://github.com/apache/iceberg/pull/12494#issuecomment-2802156779 cc @RussellSpitzer @szehon-ho @amogh-jahagirdar @nastra Could you please take a look at this PR when you have a moment? Thanks a lot! -- This is an automated message from the Apa

Re: [I] [feature request] Support Time64Type[ns] [iceberg-python]

2025-04-14 Thread via GitHub
0x26res commented on issue #1169: URL: https://github.com/apache/iceberg-python/issues/1169#issuecomment-2801964810 | Interestingly Time64Type[ns] is False when evaluating pa.types.is_timestamp(primitive). I think this is expected, time64/32 is not a timestamp but a time. ```

Re: [PR] Spec: Update row lineage requirements for upgrading tables [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12781: URL: https://github.com/apache/iceberg/pull/12781#discussion_r2042460867 ## format/spec.md: ## @@ -450,21 +448,24 @@ Within `added1`, the first added manifest, each data file's `first_row_id` follo The `first_row_id` of the EXISTING fil

Re: [I] Question Regarding DLQ Handling and Documentation [iceberg]

2025-04-14 Thread via GitHub
aaronphilip commented on issue #12759: URL: https://github.com/apache/iceberg/issues/12759#issuecomment-2801548512 > Hi [@peach12345](https://github.com/peach12345) as far as I am aware Kafka Connect Iceberg does not has DLQ support for sending the records to DLQ once the record lands to th

Re: [PR] Core: Use OutputFile.location(), InputFile.location() in Error Messages [iceberg]

2025-04-14 Thread via GitHub
nastra merged PR #12755: URL: https://github.com/apache/iceberg/pull/12755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2042524703 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] Spark 3.5 row lineage [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2042588893 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -1337,6 +1337,7 @@ public void testMergeWithBucketTransform() {

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2041801699 ## format/spec.md: ## @@ -1605,13 +1611,8 @@ All readers are required to read tables with unknown partition transforms, ignor Writing v3 metadata: * Partition Fiel

Re: [I] iceberg-rest-fixture Can't resolve HDFS HA Nameservice - UnknownHostException [iceberg]

2025-04-14 Thread via GitHub
Fokko commented on issue #12791: URL: https://github.com/apache/iceberg/issues/12791#issuecomment-2800717890 Thanks for raising the question here! Can you confirm that the files are part of the Docker image itself? Just as a heads-up, the `iceberg-rest-fixture` container is meant for

Re: [I] iceberg-rest-fixture Can't resolve HDFS HA Nameservice - UnknownHostException [iceberg]

2025-04-14 Thread via GitHub
MirerRon commented on issue #12791: URL: https://github.com/apache/iceberg/issues/12791#issuecomment-2800725439 Can confirm (I'm mounting them as a volume but they do appear in runtime). The thing is, if I disable HA on my HDFS cluster then everything works as expected so I assume it is abl

[I] Add `BoundExpression` [iceberg-cpp]

2025-04-14 Thread via GitHub
Fokko opened a new issue, #72: URL: https://github.com/apache/iceberg-cpp/issues/72 Where https://github.com/apache/iceberg-cpp/pull/58 adds the unbound expressions, I think it would be valuable also to add the BoundExpressions. When an expression is passed in from the Query Engine, we need

Re: [PR] Kafka Connect: Add delta writer support [iceberg]

2025-04-14 Thread via GitHub
ismailsimsek commented on code in PR #12070: URL: https://github.com/apache/iceberg/pull/12070#discussion_r2041568780 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -68,6 +68,8 @@ public class IcebergSinkConfig extends Abstr

Re: [PR] Spark: Support creating views via SQL [iceberg]

2025-04-14 Thread via GitHub
monologuist commented on PR #9423: URL: https://github.com/apache/iceberg/pull/9423#issuecomment-2800936417 Thank you for your patient explanation. My goal is to make Iceberg views available across multiple engines. I'll check with other communities about their plans in this regard. -- T

Re: [PR] Spark: Support creating views via SQL [iceberg]

2025-04-14 Thread via GitHub
monologuist commented on PR #9423: URL: https://github.com/apache/iceberg/pull/9423#issuecomment-2800287763 Thank you for your response! However, when I switched to Iceberg version 1.7.2, the issue remained. Did I make a mistake in my operation? -- This is an automated message from the Ap

Re: [PR] Build: Bump junit to 5.12.2 [iceberg]

2025-04-14 Thread via GitHub
nastra commented on PR #12391: URL: https://github.com/apache/iceberg/pull/12391#issuecomment-2801649102 https://github.com/apache/iceberg/pull/12786 just went in, @iProdigy can you rebase please? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] feat(table): Add json parsing of table Requirement [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on PR #393: URL: https://github.com/apache/iceberg-go/pull/393#issuecomment-2802256130 But we already support commiting with the rest API because you only need to marshal the requirements *to* JSON. You don't need to parse them. Unless you're trying to build a REST catal

Re: [PR] refactor(table): move all action names to constants [iceberg-go]

2025-04-14 Thread via GitHub
mimikwang commented on code in PR #392: URL: https://github.com/apache/iceberg-go/pull/392#discussion_r2042490232 ## table/updates.go: ## @@ -25,15 +25,25 @@ import ( ) const ( Review Comment: sounds good - will add -- This is an automated message from the Apache Git

Re: [PR] feat(table): Add json parsing of table Requirement [iceberg-go]

2025-04-14 Thread via GitHub
mimikwang commented on PR #393: URL: https://github.com/apache/iceberg-go/pull/393#issuecomment-2802248093 > This looks good, but i'm curious what the usage of this actually is For the iceberg rest api - when you make a [post request to commit a table](https://github.com/apache/iceber

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042362277 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -251,6 +266,9 @@ public IcebergSinkConfig(Map originalProps) {

Re: [PR] Kafka Connect: Add delta writer support [iceberg]

2025-04-14 Thread via GitHub
olarcherc24 commented on PR #12070: URL: https://github.com/apache/iceberg/pull/12070#issuecomment-2801973306 @bryanck while I can fully relate to your concerns, I strongly advocate for moving forward with this PR, performance considerations notwithstanding. I agree with @ajantha-bhat and t

Re: [PR] Generic Serializer and DeSerializer for control topic consumers and producers [iceberg]

2025-04-14 Thread via GitHub
kumarpritam863 commented on PR #12583: URL: https://github.com/apache/iceberg/pull/12583#issuecomment-2802130551 Thanks @bryanck for the review. Sorry I did not get, by using kafka props you mean not using consumer and producer props. -- This is an automated message from the Apache Git Se

Re: [PR] Spec: Update row lineage requirements for upgrading tables [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on code in PR #12781: URL: https://github.com/apache/iceberg/pull/12781#discussion_r2042426359 ## format/spec.md: ## @@ -450,21 +448,24 @@ Within `added1`, the first added manifest, each data file's `first_row_id` follo The `first_row_id` of the EXISTING fil

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2025-04-14 Thread via GitHub
pvary commented on PR #11497: URL: https://github.com/apache/iceberg/pull/11497#issuecomment-2801393743 Rebased based on the new API. @stevenzwu: Could you please review? Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] Spark multiple sources found for "iceberg" [iceberg]

2025-04-14 Thread via GitHub
lkindere commented on issue #12750: URL: https://github.com/apache/iceberg/issues/12750#issuecomment-2801802348 As I understand they have their own internal Iceberg source for reading Most likely this is the problem: https://docs.databricks.com/aws/en/delta/uniform as the error only o

Re: [PR] feat(iceberg): introduce remove schemas [iceberg-rust]

2025-04-14 Thread via GitHub
Xuanwo merged PR #1115: URL: https://github.com/apache/iceberg-rust/pull/1115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Remove time logical information from connect schema for spark compatibility [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12724: URL: https://github.com/apache/iceberg/pull/12724#discussion_r2042266835 ## kafka-connect/kafka-connect-transforms/src/main/java/org/apache/iceberg/connect/transforms/ConnectTimeTypeToIntegerTypeTransform.java: ## @@ -0,0 +1,316 @@ +/* + *

Re: [PR] build(deps): bump github.com/substrait-io/substrait-go/v3 from 3.9.0 to 3.9.1 in the gomod_updates group [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade merged PR #394: URL: https://github.com/apache/iceberg-go/pull/394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] I do not understand the partition error: ValueError: Could not find in old schema: 2: {field}: identity(2) [iceberg-python]

2025-04-14 Thread via GitHub
christophediprima commented on issue #1100: URL: https://github.com/apache/iceberg-python/issues/1100#issuecomment-2802083910 > Thanks for this issue, the document for `create_a_table` is still not updated and very confusing: [#create-a-table](https://py.iceberg.apache.org/api/#create-a-ta

Re: [PR] Kafka Connect: Add delta writer support [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on PR #12070: URL: https://github.com/apache/iceberg/pull/12070#issuecomment-2801858497 @danielcweeks Do you feel our stance on this evolved or should we hold off on adding this until there is more clarity on the future of equality deletes? -- This is an automated messag

Re: [PR] testing: test-container recipe for iceberg [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade merged PR #388: URL: https://github.com/apache/iceberg-go/pull/388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Patch release request CVE-2025-30065 [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on issue #12749: URL: https://github.com/apache/iceberg/issues/12749#issuecomment-2802122672 Why is an Iceberg release needed? Can't this dependency version be overridden in downstream projects? -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Patch release request CVE-2025-30065 [iceberg]

2025-04-14 Thread via GitHub
jeremysong commented on issue #12749: URL: https://github.com/apache/iceberg/issues/12749#issuecomment-2802226196 > Why is an Iceberg release needed? Can't this dependency version be overridden in downstream projects? @rdblue this is true for most of the projects. But [Iceberg Spark

Re: [I] Unable to write Map types with ParquetWriter [iceberg-rust]

2025-04-14 Thread via GitHub
jonathanc-n commented on issue #1202: URL: https://github.com/apache/iceberg-rust/issues/1202#issuecomment-2800241376 @hugokitano I'll try to clarify this in the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Spark 3.5: Use ProcedureInput for MigrateTableProcedure. [iceberg]

2025-04-14 Thread via GitHub
slfan1989 commented on PR #12782: URL: https://github.com/apache/iceberg/pull/12782#issuecomment-2800371822 @aokolnychyi @szehon-ho This is a small change. I noticed that ProcedureInput has already been used in AddFilesProcedure, and this change was implemented in #7068, so I submitted this

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

2025-04-14 Thread via GitHub
danielcweeks commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2802614392 Hey @wypoon I think we're approaching this the wrong way. Rather than defining new projects to represent hive3&4, we can actually just run the tests against those dependencies by

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
talatuyarer commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042544968 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -1224,6 +1229,9 @@ public void testListTables() { @Test public void listNamespac

Re: [PR] Remove time logical information from connect schema for spark compatibility [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12724: URL: https://github.com/apache/iceberg/pull/12724#discussion_r2042276873 ## kafka-connect/kafka-connect-transforms/src/main/java/org/apache/iceberg/connect/transforms/ConnectTimeTypeToIntegerTypeTransform.java: ## @@ -0,0 +1,316 @@ +/* + *

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
nastra commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042566581 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2636,7 +2636,7 @@ public void testTableExistsViaHEADRequest() { any()); Mo

Re: [PR] fix(catalog/rest): Fix concurrency bug in REST catalog request signing [iceberg-go]

2025-04-14 Thread via GitHub
jhump commented on PR #384: URL: https://github.com/apache/iceberg-go/pull/384#issuecomment-2802360007 > Is there any good way to test this that `-race` would find? I'll poke around. I think it will require being able to inject credentials (right now, the AWS package gets them from th

Re: [PR] fix: Don't use avro.DefaultSchemaCache [iceberg-go]

2025-04-14 Thread via GitHub
jhump commented on PR #385: URL: https://github.com/apache/iceberg-go/pull/385#issuecomment-2802368954 > Is there an example of an issue that could happen by using the default cache? Can we test for it? Good question. I think so, though it depends slightly on what the `ocf` package d

Re: [PR] Remove time logical information from connect schema for spark compatibility [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on PR #12724: URL: https://github.com/apache/iceberg/pull/12724#issuecomment-2801937060 Thanks @kumarpritam863 , having an SMT for this makes sense to me, I had a couple of comments on improving code reuse. -- This is an automated message from the Apache Git Service. To

Re: [PR] feat(table): Add json parsing of table Requirement [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on PR #393: URL: https://github.com/apache/iceberg-go/pull/393#issuecomment-2802557551 Not out of scope at all. Just considering that we may want to start splitting things out into more sub packages and such to reduce the size of the docs. But we can look into that as a

Re: [PR] fix: Don't use avro.DefaultSchemaCache [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on PR #385: URL: https://github.com/apache/iceberg-go/pull/385#issuecomment-2802561469 @jhump think it's worth trying to set up a test in this PR? I'd also be okay with putting it off to a future PR for now -- This is an automated message from the Apache Git Service. T

Re: [PR] Build: Bump junit to 5.12.2 [iceberg]

2025-04-14 Thread via GitHub
iProdigy commented on PR #12391: URL: https://github.com/apache/iceberg/pull/12391#issuecomment-2802585331 @nastra fixed the remaining build error, do you still want the PR rebased? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
kumarpritam863 commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042423856 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -251,6 +266,9 @@ public IcebergSinkConfig(Map originalPro

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
kumarpritam863 commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042422866 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -54,4 +55,7 @@ default void close(Collection closedPartitions) {

Re: [PR] doc: Clarify use of default map field name [iceberg-rust]

2025-04-14 Thread via GitHub
Xuanwo merged PR #1208: URL: https://github.com/apache/iceberg-rust/pull/1208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Generic Serializer and DeSerializer for control topic consumers and producers [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12583: URL: https://github.com/apache/iceberg/pull/12583#discussion_r2042355013 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/KafkaClientFactory.java: ## @@ -40,25 +42,88 @@ class KafkaClientFactory { this.k

Re: [PR] refactor(table): move all action names to constants [iceberg-go]

2025-04-14 Thread via GitHub
zeroshade commented on code in PR #392: URL: https://github.com/apache/iceberg-go/pull/392#discussion_r2042376016 ## table/updates.go: ## @@ -25,15 +25,25 @@ import ( ) const ( Review Comment: If we're going to make these public, we should probably stick a docstring com

Re: [PR] [1.8.x] Build: Bump Parquet from 1.15.0 to 1.15.1 (#12749) [iceberg]

2025-04-14 Thread via GitHub
rdblue commented on PR #12767: URL: https://github.com/apache/iceberg/pull/12767#issuecomment-2802083422 > Do we need to update LICENSE and NOTICE as well? If we do, please remove the version numbers from the LICENSE and NOTICE. We should not need to update them for every patch releas

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042358521 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,32 @@ */ package org.apache.iceberg.connect; -imp

Re: [PR] Decouple Committer from Kafka and Enable Custom Coordinator Election [iceberg]

2025-04-14 Thread via GitHub
bryanck commented on code in PR #12460: URL: https://github.com/apache/iceberg/pull/12460#discussion_r2042359881 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -54,4 +55,7 @@ default void close(Collection closedPartitions) { }

Re: [PR] Retry on NoSuchNamespaceException not found in rename table for rest catalog [iceberg]

2025-04-14 Thread via GitHub
nastra closed pull request #12159: Retry on NoSuchNamespaceException not found in rename table for rest catalog URL: https://github.com/apache/iceberg/pull/12159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
talatuyarer commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042637534 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2636,7 +2636,7 @@ public void testTableExistsViaHEADRequest() { any());

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
talatuyarer commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042637930 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -191,6 +192,10 @@ protected boolean supportsEmptyNamespace() { return false; }

Re: [PR] Flink: Move unlock from MemoryLock open to TestCase Before [iceberg]

2025-04-14 Thread via GitHub
pvary commented on code in PR #12793: URL: https://github.com/apache/iceberg/pull/12793#discussion_r2042815535 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java: ## @@ -62,6 +62,8 @@ void before() { Table table = create

Re: [PR] CatalogTests: Fix listNamespaces Check, Avoid Reserved Keyword, Allow Configurable Location [iceberg]

2025-04-14 Thread via GitHub
nastra commented on code in PR #12768: URL: https://github.com/apache/iceberg/pull/12768#discussion_r2042574912 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -191,6 +192,10 @@ protected boolean supportsEmptyNamespace() { return false; } + p

Re: [PR] Introduce AuthManager [iceberg-python]

2025-04-14 Thread via GitHub
Fokko commented on code in PR #1908: URL: https://github.com/apache/iceberg-python/pull/1908#discussion_r2042820333 ## pyiceberg/catalog/rest/auth.py: ## @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

  1   2   >