Re: [PR] feat: implement initial MemoryCatalog functionality with namespace and table support [iceberg-cpp]

wgtmac commented on code in PR #80: URL: https://github.com/apache/iceberg-cpp/pull/80#discussion_r2059433981 ## src/iceberg/CMakeLists.txt: ## @@ -19,6 +19,7 @@ set(ICEBERG_INCLUDES "$" "$") set(ICEBERG_SOURCES arrow_c_data_internal.cc +catalog/

Re: [PR] [SPARK] Fix add_files type conversion exception and incorrect partition value when handling null partitions [iceberg]

geruh commented on PR #12886: URL: https://github.com/apache/iceberg/pull/12886#issuecomment-2829506425 Interesting, so only versions 3.4, 3.5 that have this exception? But yeah we should just duplicate the cases for each affected versions. > Not sure if it important enough to backpo

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

aihuaxu commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059559300 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -879,6 +896,405 @@ public void testMixedRecords() throws IOException { Variant

Re: [I] When iceberg.tables.auto-create-enabled is set true the Iceberg Kafka Connect Sink will try to write to local database of Hive metastore instead of the bucket on object storage [iceberg]

lk-1984 commented on issue #12648: URL: https://github.com/apache/iceberg/issues/12648#issuecomment-2829401194 Anyone planning to fix this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

wypoon commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2829379097 @danielcweeks I rebased on main. When you find the time, please open a PR against my branch or otherwise post the gradle changes you're suggesting. Thanks. -- This is an automated m

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059551800 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,95 @@ +/* + * Li

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059536198 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java: ## @@ -317,28 +316,6 @@ public void testConflictingColumns()

Re: [PR] refactor(s3tables): avoid misleading FileIO::from_path [iceberg-rust]

liurenjie1024 merged PR #1240: URL: https://github.com/apache/iceberg-rust/pull/1240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Introduce scheduler for sqllogictests [iceberg-rust]

liurenjie1024 commented on code in PR #1244: URL: https://github.com/apache/iceberg-rust/pull/1244#discussion_r2059511627 ## crates/sqllogictest/src/schedule.rs: ## @@ -0,0 +1,105 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [I] Add `CatalogLoader` trait. [iceberg-rust]

liurenjie1024 commented on issue #1254: URL: https://github.com/apache/iceberg-rust/issues/1254#issuecomment-2829286621 > For `CatalogLoader` trait, how about we have a `CatalogConfig` like: > > pub struct CatalogConfig { > name: String, > uri: String, > warehouse:

Re: [PR] feat: implement initial MemoryCatalog functionality with namespace and table support [iceberg-cpp]

zhjwpku commented on code in PR #80: URL: https://github.com/apache/iceberg-cpp/pull/80#discussion_r2059495353 ## src/iceberg/catalog/memory_catalog.cc: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreem

Re: [PR] Add catalog builder trait [iceberg-rust]

Xuanwo commented on PR #1261: URL: https://github.com/apache/iceberg-rust/pull/1261#issuecomment-2829280470 I have a new proposal in https://github.com/apache/iceberg-rust/issues/1254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Add catalog builder trait [iceberg-rust]

liurenjie1024 commented on PR #1261: URL: https://github.com/apache/iceberg-rust/pull/1261#issuecomment-2829279257 cc @Xuanwo @sdd @c-thiel @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Add catalog builder trait [iceberg-rust]

liurenjie1024 commented on PR #1261: URL: https://github.com/apache/iceberg-rust/pull/1261#issuecomment-2829279034 I tried to use object safety trait, but found a lot limitation which will make the code difficult to use, so I rollback to this version. For a more complete example, see #1231

Re: [I] Add `CatalogLoader` trait. [iceberg-rust]

Xuanwo commented on issue #1254: URL: https://github.com/apache/iceberg-rust/issues/1254#issuecomment-2829275404 For `CatalogLoader` trait, how about we have a `CatalogConfig` like: ```rust pub struct CatalogConfig { name: String, uri: String, warehouse: String,

Re: [PR] Build: Retry flaky test [iceberg]

manuzhang closed pull request #12707: Build: Retry flaky test URL: https://github.com/apache/iceberg/pull/12707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Flink supports clean orphan files [iceberg]

sunxiaojian closed issue #12674: Flink supports clean orphan files URL: https://github.com/apache/iceberg/issues/12674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Flink: supports clean orphan files [iceberg]

sunxiaojian closed pull request #12754: Flink: supports clean orphan files URL: https://github.com/apache/iceberg/pull/12754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Flink: supports clean orphan files [iceberg]

sunxiaojian commented on PR #12754: URL: https://github.com/apache/iceberg/pull/12754#issuecomment-2829236490 @pvary @pvary thanks for your review， closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Spark 3.5: Disable executor cache for delete files in RewriteDataFilesSparkAction [iceberg]

ebyhr commented on code in PR #12893: URL: https://github.com/apache/iceberg/pull/12893#discussion_r2059438422 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -79,6 +79,11 @@ private SparkSQLProperties() {} public static final String

Re: [PR] Spark 3.5: Disable executor cache for delete files in RewriteDataFilesSparkAction [iceberg]

manuzhang commented on code in PR #12893: URL: https://github.com/apache/iceberg/pull/12893#discussion_r2059431464 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -109,6 +110,7 @@ public class RewriteDataFilesSparkAction

[PR] [catalog] Expose TableCommit construction to public [iceberg-rust]

dentiny opened a new pull request, #1252: URL: https://github.com/apache/iceberg-rust/pull/1252 ## What changes are included in this PR? In this PR, I expose `TableCommit` builder to public, so I could create the instance outside of the crate. Some context why I need this chang

Re: [I] Implement list_views Method for hive catalog [iceberg-python]

github-actions[bot] commented on issue #1240: URL: https://github.com/apache/iceberg-python/issues/1240#issuecomment-2829113425 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] feat(puffin): Add reader and writer [iceberg-rust]

fqaiser94 closed pull request #714: feat(puffin): Add reader and writer URL: https://github.com/apache/iceberg-rust/pull/714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat(puffin): Add reader and writer [iceberg-rust]

fqaiser94 commented on PR #714: URL: https://github.com/apache/iceberg-rust/pull/714#issuecomment-2829113056 Can close this draft now as all the code has been merged as separate smaller PRs. See https://github.com/apache/iceberg-rust/issues/744 for more details. -- This is an automat

Re: [PR] Core: Fix locationProvider implementation for SerializableTable [iceberg]

github-actions[bot] closed pull request #12564: Core: Fix locationProvider implementation for SerializableTable URL: https://github.com/apache/iceberg/pull/12564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Core: Fix locationProvider implementation for SerializableTable [iceberg]

github-actions[bot] commented on PR #12564: URL: https://github.com/apache/iceberg/pull/12564#issuecomment-2829110859 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

github-actions[bot] commented on PR #12449: URL: https://github.com/apache/iceberg/pull/12449#issuecomment-2829110819 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Select for rewriting the files belonging to old partitioning schemes [iceberg]

github-actions[bot] commented on PR #12083: URL: https://github.com/apache/iceberg/pull/12083#issuecomment-2829110775 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] RemoveDanglingDeleteFiles [iceberg]

szehon-ho commented on issue #6126: URL: https://github.com/apache/iceberg/issues/6126#issuecomment-2829109090 > When using rewitePositionDeletes, should keeping this as true? Yes (if I understand correctly the question) -- This is an automated message from the Apache Git Service. T

Re: [PR] feat(catalog/glue): Add SetTableProperties for glue catalog [iceberg-go]

lliangyu-lin commented on code in PR #395: URL: https://github.com/apache/iceberg-go/pull/395#discussion_r2059393092 ## catalog/glue/glue.go: ## @@ -330,6 +331,53 @@ func (c *Catalog) RegisterTable(ctx context.Context, identifier table.Identifier return c.LoadTable(ctx,

Re: [PR] [SPARK] Fix add_files type conversion exception and incorrect partition value when handling null partitions [iceberg]

hariuserx commented on PR #12886: URL: https://github.com/apache/iceberg/pull/12886#issuecomment-2829103358 > Nice @hariuserx! Now the util will preserve the null partition value. Are we seeing this happen with the other spark versions or should we backport this fix as well? Yeah. It

[PR] feat: Add commit table support for Glue Catalog [iceberg-go]

lliangyu-lin opened a new pull request, #403: URL: https://github.com/apache/iceberg-go/pull/403 ### Description * Implement `CommitTable` defined in the catalog interface * It will takes in `updates` and apply them on the base table and create a staging table * The table tr

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059384245 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -0,0 +1,443 @@ +/* + * License

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059375632 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -0,0 +1,436 @@ +/* + * License

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059375632 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -0,0 +1,436 @@ +/* + * License

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059375632 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -0,0 +1,436 @@ +/* + * License

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on PR #12841: URL: https://github.com/apache/iceberg/pull/12841#issuecomment-2829068292 Thanks for merging, @RussellSpitzer! And thanks to all the reviewers, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059372436 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -879,6 +896,405 @@ public void testMixedRecords() throws IOException { VariantT

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059371968 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -879,6 +896,405 @@ public void testMixedRecords() throws IOException { VariantT

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059371622 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -879,6 +896,405 @@ public void testMixedRecords() throws IOException { VariantT

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059371134 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -879,6 +896,405 @@ public void testMixedRecords() throws IOException { VariantT

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059370149 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -177,7 +177,19 @@ public static

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059369475 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -900,6 +1316,102 @@ private static GenericRecord record(GroupType type, Map fields)

Re: [PR] Spark 3.5 row lineage [iceberg]

amogh-jahagirdar commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059363985 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -0,0 +1,454 @@ +/* + * License

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer merged PR #12841: URL: https://github.com/apache/iceberg/pull/12841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on PR #12841: URL: https://github.com/apache/iceberg/pull/12841#issuecomment-2829059296 Merged! Thank you @rdblue for tightening up the spec here. Long thanks to all the reviewers @adutra @manuzhang @wmoustafa @nastra @danielcweeks @amog

[I] Feature request: Allow pagination on list operations [iceberg-rust]

dentiny opened a new issue, #1251: URL: https://github.com/apache/iceberg-rust/issues/1251 ### Is your feature request related to a problem or challenge? It's possible to have large number of namespace and tables, with query taking long time listing these large entities. I'm wondering

Re: [PR] SPARK: Remove dependency on hadoop's filesystem class from remove orphan files [iceberg]

fuzing commented on PR #12254: URL: https://github.com/apache/iceberg/pull/12254#issuecomment-2829035039 @RussellSpitzer - we ran further tests today and are pleased to report that for S3 this PR appears to work correctly with the following caveat: - The files deleted (or proposed del

[PR] [easy] [discussion] Add more error types [iceberg-rust]

dentiny opened a new pull request, #1250: URL: https://github.com/apache/iceberg-rust/pull/1250 ## Which issue does this PR close? Related to https://github.com/apache/iceberg-rust/issues/1249 ## What changes are included in this PR? In this PR, I introduce two new error

[I] [Feature request] Add more error types for iceberg [iceberg-rust]

dentiny opened a new issue, #1249: URL: https://github.com/apache/iceberg-rust/issues/1249 ### Is your feature request related to a problem or challenge? As of now, our error status type is pretty limited: https://github.com/apache/iceberg-rust/blob/c34982a89aa426cdb7e529f8c5c866f846b

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059348242 ## format/spec.md: ## @@ -266,7 +266,18 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `writ

Re: [PR] fix upsert with complex types [iceberg-python]

kevinjqliu commented on PR #1949: URL: https://github.com/apache/iceberg-python/pull/1949#issuecomment-2828993461 closing in favor of #1878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] fix upsert with complex types [iceberg-python]

kevinjqliu closed pull request #1949: fix upsert with complex types URL: https://github.com/apache/iceberg-python/pull/1949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[PR] [catalog] Fix namespace creation error status [iceberg-rust]

dentiny opened a new pull request, #1248: URL: https://github.com/apache/iceberg-rust/pull/1248 ## Which issue does this PR close? - Closes #1247 ## What changes are included in this PR? According to the [openapi spec](https://github.com/apache/iceberg-rust/issues/1247)

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

kevinjqliu commented on PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#issuecomment-2828992965 Thanks @koenvo for the PR and @Fokko for the review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Upsert with list type not supported [iceberg-python]

kevinjqliu closed issue #1711: Upsert with list type not supported URL: https://github.com/apache/iceberg-python/issues/1711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

kevinjqliu merged PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

[I] [Bug] Error HTTP status handling [iceberg-rust]

dentiny opened a new issue, #1247: URL: https://github.com/apache/iceberg-rust/issues/1247 ### Apache Iceberg Rust version 0.4.0 (latest version) ### Describe the bug Reading through the code, when we try to create a namespace and issue a POST request, we propagate `Unex

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059311736 ## core/src/test/java/org/apache/iceberg/variants/TestValueArray.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059311125 ## core/src/test/java/org/apache/iceberg/variants/TestValueArray.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2059310791 ## core/src/test/java/org/apache/iceberg/variants/TestValueArray.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Add timestamp_ns, time and UUID types for Variant [iceberg]

rdblue merged PR #12682: URL: https://github.com/apache/iceberg/pull/12682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059304550 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `write-defaul

Re: [PR] [SPARK] Fix add_files type conversion exception and incorrect partition value when handling null partitions [iceberg]

geruh commented on PR #12886: URL: https://github.com/apache/iceberg/pull/12886#issuecomment-2828954798 Nice @hariuserx! Now the util will preserve the null partition value. Are we seeing this happen with the other spark versions or should we backport this fix as well? -- This is an auto

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059291495 ## core/src/main/java/org/apache/iceberg/variants/ShreddedObject.java: ## @@ -61,7 +61,7 @@ VariantMetadata metadata() { } private Set nameSet() { -Set nam

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2059257480 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -2011,6 +2011,40 @@ public void testZOrderRewriteWith

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059276361 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `writ

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059275200 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `writ

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059225620 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `writ

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2059254400 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -2011,6 +2011,40 @@ public void testZOrderRewriteWith

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059271656 ## core/src/main/java/org/apache/iceberg/variants/ShreddedObject.java: ## @@ -61,7 +61,7 @@ VariantMetadata metadata() { } private Set nameSet() { -

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059260787 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `write-defaul

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2059258772 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,37 @@ private Builder doExecuteWithPart

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2059258772 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,37 @@ private Builder doExecuteWithPart

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059257190 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `write-defaul

Re: [I] Nessie should throw a NoSuchNamespaceException when listing a non-existing namespace [iceberg]

akshatmardia commented on issue #12875: URL: https://github.com/apache/iceberg/issues/12875#issuecomment-2828882562 I can work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2059241409 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPart

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

RussellSpitzer commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059237372 ## format/spec.md: ## @@ -266,7 +266,11 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `writ

Re: [PR] Spark 3.5: Disable executor cache for delete files in RewriteDataFilesSparkAction [iceberg]

anuragmantri commented on PR #12893: URL: https://github.com/apache/iceberg/pull/12893#issuecomment-2828869934 retest please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

kevinjqliu commented on code in PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#discussion_r2059226527 ## tests/table/test_upsert.py: ## @@ -511,6 +511,163 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: tbl.upsert(df) +def

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

koenvo commented on code in PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#discussion_r2059189751 ## pyiceberg/table/upsert_util.py: ## @@ -71,25 +72,59 @@ def get_rows_to_update(source_table: pa.Table, target_table: pa.Table, join_cols # When the ta

Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]

GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO` More context why I want to have the directory support, I'm implementing a filesystem catalog feature for two purpose: - Hermetic unit test + bringing rest docker image is a little painful, but I

Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]

GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO` More context why I want to have the directory support: I'm implementing a filesystem catalog feature for two purpose: - Hermetic unit test + bringing rest docker image is a little painful, but I h

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

koenvo commented on code in PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#discussion_r2059201230 ## tests/table/test_upsert.py: ## @@ -511,6 +511,96 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: tbl.upsert(df) +def test

Re: [PR] Spec: Avoid struct field conflicts in default values [iceberg]

rdblue commented on code in PR #12841: URL: https://github.com/apache/iceberg/pull/12841#discussion_r2059186760 ## format/spec.md: ## @@ -266,7 +266,9 @@ The `initial-default` is set only when a field is added to an existing schema. T The `initial-default` and `write-default

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

kevinjqliu commented on code in PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#discussion_r2059186918 ## tests/table/test_upsert.py: ## @@ -511,6 +511,96 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: tbl.upsert(df) +def

Re: [PR] feat: `validation_history` and `ancestors_between` [iceberg-python]

jayceslesar commented on code in PR #1935: URL: https://github.com/apache/iceberg-python/pull/1935#discussion_r2059165568 ## pyiceberg/table/update/validate.py: ## @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Fallback for upsert when arrow cannot compare source rows with target rows [iceberg-python]

Fokko commented on code in PR #1878: URL: https://github.com/apache/iceberg-python/pull/1878#discussion_r2059170856 ## pyiceberg/table/upsert_util.py: ## @@ -71,25 +72,59 @@ def get_rows_to_update(source_table: pa.Table, target_table: pa.Table, join_cols # When the tar

Re: [PR] Add timestamp_ns, time and UUID types for Variant [iceberg]

aihuaxu commented on code in PR #12682: URL: https://github.com/apache/iceberg/pull/12682#discussion_r2059168065 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetVariantVisitor.java: ## @@ -282,4 +284,18 @@ private static boolean isBinary(Type type) { return type.

Re: [PR] Feature: Write to branches [iceberg-python]

Fokko commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2828736306 Thanks @dbuades for pinging me, and sorry for letting this one linger for so long. Let me review this tomorrow morning. @vinjai Do you have time to resolve the conflicts by any c

Re: [PR] Feat/simplify upsert [iceberg-python]

koenvo closed pull request #1947: Feat/simplify upsert URL: https://github.com/apache/iceberg-python/pull/1947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Doc: Hint implications on use write.data.path and orphan removal together [iceberg]

Fokko commented on PR #12890: URL: https://github.com/apache/iceberg/pull/12890#issuecomment-2828724788 Thanks for the context @dramaticlly. After thinking about a bit more, I think it makes more sense to put this in the orphan files cleanup routine? Including a warning that it will delete

Re: [PR] Spark 3.5: Disable executor cache for delete files in RewriteDataFilesSparkAction [iceberg]

anuragmantri commented on PR #12893: URL: https://github.com/apache/iceberg/pull/12893#issuecomment-2828714671 @aokolnychyi @amogh-jahagirdar - Could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Introduce scheduler for sqllogictests [iceberg-rust]

jonathanc-n commented on code in PR #1244: URL: https://github.com/apache/iceberg-rust/pull/1244#discussion_r2059115379 ## crates/sqllogictest/src/schedule.rs: ## @@ -0,0 +1,105 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] feat: `validation_history` and `ancestors_between` [iceberg-python]

Fokko commented on code in PR #1935: URL: https://github.com/apache/iceberg-python/pull/1935#discussion_r2059094364 ## pyiceberg/table/update/validate.py: ## @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Spark 3.5 row lineage [iceberg]

rdblue commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059093631 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,95 @@ +/* + * Licensed to

Re: [PR] Spark 3.5 row lineage [iceberg]

rdblue commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059092298 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,95 @@ +/* + * Licensed to

Re: [PR] Spark 3.5 row lineage [iceberg]

rdblue commented on code in PR #12736: URL: https://github.com/apache/iceberg/pull/12736#discussion_r2059090462 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTableForRowLineage.scala: ## @@ -0,0 +1,95 @@ +/* + * Licensed to

Re: [PR] feat: `validation_history` and `ancestors_between` [iceberg-python]

Fokko commented on code in PR #1935: URL: https://github.com/apache/iceberg-python/pull/1935#discussion_r2059089720 ## pyiceberg/table/update/validate.py: ## @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

  1   2   >