Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705020634 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -28,51 +28,79 @@ public class TestFormatVersions extends TestBase { @Parameters(name = "for

Re: [I] Structured streaming writes to partitioned table fails when spark.sql.extensions is set to IcebergSparkSessionExtensions [iceberg]

wfxxh commented on issue #7226: URL: https://github.com/apache/iceberg/issues/7226#issuecomment-2270542484 spark version: 3.3.3 , iceberg version: 1.4.3 ,same stack trace for me. my table is: > CREATE TABLE iceberg_catalog.doi.doi_log ( date_time TIMESTAMP, ip STRING,

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705024826 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1451,50 +1457,67 @@ public void testCreateV2MetadataThroughTableProperty() { .doesNotC

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705025435 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1451,50 +1457,67 @@ public void testCreateV2MetadataThroughTableProperty() { .doesNotC

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705027285 ## core/src/test/java/org/apache/iceberg/TestUpdateRequirements.java: ## @@ -198,11 +201,26 @@ public void upgradeFormatVersion() { assertTableUUID(requirements);

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705027632 ## core/src/test/java/org/apache/iceberg/TestUpdateRequirements.java: ## @@ -198,11 +201,26 @@ public void upgradeFormatVersion() { assertTableUUID(requirements);

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705028854 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705035276 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705038426 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705039041 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705040035 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705040838 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.

Re: [I] Quick start docker-compose demo doesn't work [iceberg]

nastra commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-2270572351 @awongCM please open a separate issue with details about the issue you're running into and also the versions of the different docker images you're using -- This is an automated mess

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

pvary commented on code in PR #10881: URL: https://github.com/apache/iceberg/pull/10881#discussion_r1705051705 ## flink/v1.20/flink/src/jmh/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitionerBenchmark.java: ## @@ -140,8 +139,7 @@ private static String randomString(Str

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

pvary commented on code in PR #10881: URL: https://github.com/apache/iceberg/pull/10881#discussion_r1705052375 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java: ## @@ -401,7 +401,6 @@ void createIcebergTable(ObjectPath tablePath, ResolvedCatalogTabl

Re: [I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

Fokko commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2270597684 @djouallah Thanks for raising this. To clarify, does the `final.arrow()` cause an OOM, or the `.append` operation? -- This is an automated message from the Apache Git Service

Re: [PR] Bump pypa/cibuildwheel from 2.19.2 to 2.20.0 [iceberg-python]

Fokko merged PR #999: URL: https://github.com/apache/iceberg-python/pull/999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270617271 We will need https://github.com/apache/opendal/issues/4967 for this. I'm working on it now. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

djouallah commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2270625415 Append operation cause OOM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

chenzl25 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270665390 > We will need [apache/opendal#4967](https://github.com/apache/opendal/issues/4967) for this. I'm working on it now. Which opendal's version do we need to bump into? --

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270668680 > Which opendal's version do we need to bump into? I'm guessing it will be included in our next release `0.49`. -- This is an automated message from the Apache Git Service.

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

chenzl25 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270672894 I have already verified that using a global HTTP client works in this issue and the performance improvement is impressive (about 5 times faster: from 500+ms to 100+ms). http

Re: [PR] OpenAPI: Add RemovePartitionSpecsUpdate REST update type [iceberg]

advancedxy commented on PR #10846: URL: https://github.com/apache/iceberg/pull/10846#issuecomment-2270674733 > I think we should probably wait for that to go in before proceeding here, just so the community is on the same page. Of course. -- This is an automated message from the Ap

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270676235 Also, cc @sdd, who is focusing on the iceberg benchmark now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Core: Make namespace separator configurable [iceberg]

nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1705139878 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,26 @@ public static String encodeNamespace(Namespace ns) { * @return a namespace

Re: [PR] Core: Make namespace separator configurable [iceberg]

nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1705142884 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -116,6 +116,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog privat

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

c-thiel commented on PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#issuecomment-2270712857 > I have a few `ToDo`s in the code. Most of them are for tests in other modules that would fail with proper partition binding. > > If someone has insights on the following, it wo

Re: [PR] Core: Make namespace separator configurable [iceberg]

nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1705174451 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -116,6 +116,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog privat

Re: [I] Use Min, Max, and NumOfNulls from Manifest Files for Spark Column Stats [iceberg]

jeesou commented on issue #10791: URL: https://github.com/apache/iceberg/issues/10791#issuecomment-2270769479 Hi @huaxingao, could you please help us with the timeline for the subsequent PR to add the min, max, and numOfNulls into the puffin file? -- This is an automated message from the

Re: [PR] Build: Bump org.testcontainers:testcontainers from 1.20.0 to 1.20.1 [iceberg]

nastra merged PR #10865: URL: https://github.com/apache/iceberg/pull/10865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r170524 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705245091 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java: ## @@ -60,6 +61,14 @@ private FlinkWriteOptions() {} public static final ConfigOp

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705245993 ## docs/docs/flink-writes.md: ## @@ -262,6 +262,91 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ Check out all the options here: [write-options]

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1705248284 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1705248739 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1705249312 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705247759 ## docs/docs/flink-writes.md: ## @@ -262,6 +262,91 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ Check out all the options here: [write-options]

[I] Cannot specify "file://" uri or direct local file location for warehouse in Windows [iceberg-python]

rfung777 opened a new issue, #1005: URL: https://github.com/apache/iceberg-python/issues/1005 ### Apache Iceberg version 0.7.0 (latest release) ### Please describe the bug 🐞 When running PyIceberg in Windows, and using local file system as warehouse location, we cannot s

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705275468 ## docs/docs/flink-writes.md: ## @@ -262,6 +262,91 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ Check out all the options here: [write-options]

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705279951 ## docs/docs/flink-writes.md: ## @@ -262,6 +262,91 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ Check out all the options here: [write-options]

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705281270 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java: ## @@ -233,15 +239,56 @@ public Builder flinkConf(ReadableConfig config) { * @ret

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705286597 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java: ## @@ -548,21 +599,46 @@ private DataStream distributeDataStream( }

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on code in PR #10859: URL: https://github.com/apache/iceberg/pull/10859#discussion_r1705298661 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkDistributionMode.java: ## @@ -177,4 +185,288 @@ public void testOverrideWriteConfi

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

pvary commented on PR #10859: URL: https://github.com/apache/iceberg/pull/10859#issuecomment-2270944191 @rodmeneses: This will effect your PR as well. Please sync with @stevenzwu about the order of the commits -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Flink: infer source parallelism for FLIP-27 source in batch execution mode [iceberg]

pvary commented on code in PR #10832: URL: https://github.com/apache/iceberg/pull/10832#discussion_r1705305624 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -79,6 +80,12 @@ public class TestIcebergSourceWit

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

sdd commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270982857 I also have some local code that reuses the same OpenDAL operator rather than creating a new one each time. I'd not submitted it yet as I wasn't sure of the validity of doing that in

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270991646 > I also have some local code that reuses the same OpenDAL operator rather than creating a new one each time. I'd not submitted it yet as I wasn't sure of the validity of doing tha

Re: [PR] Concurrent table scans [iceberg-rust]

sdd commented on PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2271017314 Sorry guys, been off-grid for a few days. I've addressed all of your suggestions @liurenjie1024, thanks. > > Thanks @sdd! That's awesome! BTW, can we integrate the performance test

Re: [PR] Manifest list encryption [iceberg]

ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1705350346 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -159,4 +159,6 @@ private CatalogProperties() {} public static final String ENCRYPTION_KM

Re: [PR] Data: Add a util to read write partition stats [iceberg]

ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1705461680 ## data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Data: Add a util to read write partition stats [iceberg]

ajantha-bhat commented on PR #10176: URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2271191550 @aokolnychyi , @RussellSpitzer, @lirui-apache: Please take another look at the PR. I have addressed all the comments. -- This is an automated message from the Apache Git Servi

[PR] Completed modifying the document file. [iceberg]

DongilMin opened a new pull request, #10884: URL: https://github.com/apache/iceberg/pull/10884 I modified some of the grammar more accurately. I also modified the contents of some files to fit the markdown format. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Bump sqlalchemy from 2.0.31 to 2.0.32 [iceberg-python]

Fokko merged PR #1003: URL: https://github.com/apache/iceberg-python/pull/1003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[I] Unable to create iceberg tables in Databrics [iceberg]

sathwick9 opened a new issue, #10885: URL: https://github.com/apache/iceberg/issues/10885 Hello, I am using databricks runtime is DBR 14.3 LTS Spark 3.5.0. Scala 2.12. And using iceberg iceberg_spark_runtime 3_5_2 12 16 0.jar. Is it correct version i am using, because when i installed

Re: [PR] Bump cython from 3.0.10 to 3.0.11 [iceberg-python]

Fokko merged PR #1002: URL: https://github.com/apache/iceberg-python/pull/1002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Bump mkdocs-material from 9.5.30 to 9.5.31 [iceberg-python]

Fokko merged PR #1001: URL: https://github.com/apache/iceberg-python/pull/1001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [I] Unable to create iceberg tables in Databrics [iceberg]

nastra closed issue #10885: Unable to create iceberg tables in Databrics URL: https://github.com/apache/iceberg/issues/10885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

ndrluis commented on PR #997: URL: https://github.com/apache/iceberg-python/pull/997#issuecomment-2271271527 Thank you, @grobgl. Could you please double-check your test implementation? I removed the fix and the test didn't break. -- This is an automated message from the Apache Git Service

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

grobgl commented on PR #997: URL: https://github.com/apache/iceberg-python/pull/997#issuecomment-2271295554 > Thank you, @grobgl. Could you please double-check your test implementation? I removed the fix and the test didn't break. Removing the fix does break the test for me. Possibly,

Re: [PR] Data: Add a util to read write partition stats [iceberg]

lirui-apache commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1705560828 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

findepi commented on code in PR #10886: URL: https://github.com/apache/iceberg/pull/10886#discussion_r1705567611 ## .baseline/checkstyle/checkstyle.xml: ## @@ -414,6 +414,11 @@ + + Review Comment: current pattern

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

findepi commented on code in PR #10886: URL: https://github.com/apache/iceberg/pull/10886#discussion_r1705578042 ## .baseline/checkstyle/checkstyle.xml: ## @@ -414,6 +414,11 @@ + + Review Comment: can you give https:

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

nastra commented on code in PR #10886: URL: https://github.com/apache/iceberg/pull/10886#discussion_r1705585282 ## .baseline/checkstyle/checkstyle.xml: ## @@ -414,6 +414,11 @@ + + Review Comment: I don't think we want

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

Xuanwo commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1705592814 ## open-api/rest-catalog-open-api.yaml: ## @@ -2747,6 +2747,54 @@ components: uuid: type: string +AzureCredentials: + type: object +

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

nastra commented on code in PR #10886: URL: https://github.com/apache/iceberg/pull/10886#discussion_r1705611373 ## .baseline/checkstyle/checkstyle.xml: ## @@ -414,6 +414,11 @@ + + Review Comment: @findepi `LITERAL_ASS

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1705625534 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

ndrluis commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2271413306 I believe the issue is not that it returns an empty result, because if you filter for a value that does not exist, like `dataset_5`, it will not return an error. However, if you

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705638405 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/MetricsReporterFactoryForTests.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apach

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

nastra commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1705637574 ## common/src/main/java/org/apache/iceberg/common/DynConstructors.java: ## @@ -82,14 +76,6 @@ public R invoke(Object target, Object... args) { return (R) newIn

Re: [I] pyiceberg always return false for catalog.table_exists when used with Polaris catalog [iceberg-python]

ndrluis commented on issue #1006: URL: https://github.com/apache/iceberg-python/issues/1006#issuecomment-2271430444 Hello @djouallah, I did a test with the Tabular catalog and table_exists works as expected. So I believe this is a bug in the Polaris catalog. -- This is an automated messa

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

nastra commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1705642075 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -127,8 +125,9 @@ public String toString() { /** Singleton {@link UnboundMethod}, performs

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

ndrluis commented on PR #997: URL: https://github.com/apache/iceberg-python/pull/997#issuecomment-2271435500 That's strange because I'm running your branch on my machine, so we shouldn't be seeing different behavior. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

nastra commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1705643362 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -71,7 +69,7 @@ public R invokeChecked(Object target, Object... args) throws Exception {

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705645226 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java: ## @@ -0,0 +1,620 @@ +/* + * Licensed to the Apache Software F

Re: [PR] API: Define RepairManifests action interface [iceberg]

nastra commented on code in PR #10784: URL: https://github.com/apache/iceberg/pull/10784#discussion_r1705661534 ## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

ndrluis commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2271461297 @grobgl, I tested your branch with the example provided here, and your solution fixed the problem. I'll double-check why the test you wrote in your pull request is not failing o

Re: [PR] API: Define RepairManifests action interface [iceberg]

nastra commented on code in PR #10784: URL: https://github.com/apache/iceberg/pull/10784#discussion_r1705661534 ## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] fix: complete miss attribute for map && list in avro schema [iceberg-rust]

ZENOTME commented on PR #411: URL: https://github.com/apache/iceberg-rust/pull/411#issuecomment-2271476976 Thanks for work on the new Avro by @Fokko! I have updated Avro to 0.17 and unit test. PTAL cc @liurenjie1024 @Xuanwo @Fokko -- This is an automated message from the Apache Git Servic

Re: [PR] Simplify PrimitiveLiteral [iceberg-rust]

ZENOTME commented on code in PR #502: URL: https://github.com/apache/iceberg-rust/pull/502#discussion_r1705677244 ## crates/iceberg/src/spec/values.rs: ## @@ -65,24 +65,14 @@ pub enum PrimitiveLiteral { Float(OrderedFloat), /// Stored as 8-byte little-endian Doubl

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

Fokko commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1705687319 ## open-api/rest-catalog-open-api.yaml: ## @@ -2747,6 +2747,81 @@ components: uuid: type: string +ADLSCredentials: + type: object +

[I] DOCS: Improve Documentation on Write Support [iceberg-python]

sungwy opened a new issue, #1008: URL: https://github.com/apache/iceberg-python/issues/1008 ### Feature Request / Improvement We currently have Write support through two modes: 1. through the Table API 2. through the Transaction API We also have support for different modes

Re: [I] Merge into / Upsert [iceberg-python]

sungwy commented on issue #402: URL: https://github.com/apache/iceberg-python/issues/402#issuecomment-2271507538 Hi @ev2900 - would using the `overwrite` feature by specifying the boolean expression on which to upsert work for your use case? https://github.com/apache/iceberg-python/b

Re: [I] Merge into / Upsert [iceberg-python]

sungwy commented on issue #402: URL: https://github.com/apache/iceberg-python/issues/402#issuecomment-2271531980 @ev2900 agreed :) I've added that Issue above (#1008) to address that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

sungwy commented on PR #997: URL: https://github.com/apache/iceberg-python/pull/997#issuecomment-2271552739 Hi @Fokko and @ndrluis thank you for your reviews, and @grobgl for reporting this issue and getting the fix ready so quickly! I've ran the CI, but it looks like it's failing the

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705727844 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,324 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705729180 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java: ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705732746 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java: ## @@ -0,0 +1,620 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

stevenzwu commented on PR #10859: URL: https://github.com/apache/iceberg/pull/10859#issuecomment-2271588597 > @rodmeneses: This will effect your PR as well. Please sync with @stevenzwu about the order of the commits I don't think we should worry about the order. we can integrate the r

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

stevenzwu merged PR #10881: URL: https://github.com/apache/iceberg/pull/10881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

stevenzwu commented on PR #10881: URL: https://github.com/apache/iceberg/pull/10881#issuecomment-2271607408 thanks @pvary for the review. Did `rebase and merge` to preserve the Individual commits -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

sungwy merged PR #997: URL: https://github.com/apache/iceberg-python/pull/997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

sungwy closed issue #992: table.scan queries failing sometimes when result is empty URL: https://github.com/apache/iceberg-python/issues/992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

sungwy closed issue #992: table.scan queries failing sometimes when result is empty URL: https://github.com/apache/iceberg-python/issues/992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

nastra commented on PR #10886: URL: https://github.com/apache/iceberg/pull/10886#issuecomment-2271669505 thanks for the reviews @findepi @bryanck @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Build: Add checkstyle rule to ban assert usage [iceberg]

nastra merged PR #10886: URL: https://github.com/apache/iceberg/pull/10886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Bump moto from 5.0.11 to 5.0.12 [iceberg-python]

Fokko merged PR #1000: URL: https://github.com/apache/iceberg-python/pull/1000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Update metadata-log for non-rest catalogs [iceberg-python]

kevinjqliu commented on PR #977: URL: https://github.com/apache/iceberg-python/pull/977#issuecomment-2271708010 Waiting for CI to run. r? @sungwy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Add ndrluis as collaborator [iceberg-python]

sungwy merged PR #1009: URL: https://github.com/apache/iceberg-python/pull/1009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705906126 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1705910873 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -23,56 +23,100 @@ import java.util.Arrays; import java.util.List; +import java.util.

  1   2   3   >