Re: [I] Issue in Reading Iceberg tables in Nessie + Minio using Pyiceberg [iceberg-python]

2025-01-22 Thread via GitHub
Fokko commented on issue #1560: URL: https://github.com/apache/iceberg-python/issues/1560#issuecomment-2606940743 @heman026 Thanks for raising this issue. I'm not super familiar with Nessie, but I do notice that the `warehouse` configuration should be an s3 path: `s3a://iceberg-datalake/`

Re: [I] Enhance GCSFileIO with Bulk and Prefix Operations [iceberg]

2025-01-22 Thread via GitHub
Fokko closed issue #12041: Enhance GCSFileIO with Bulk and Prefix Operations URL: https://github.com/apache/iceberg/issues/12041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] ExponentialHttpRequestRetryStrategy Enhancement [iceberg]

2025-01-22 Thread via GitHub
Fokko closed issue #12042: ExponentialHttpRequestRetryStrategy Enhancement URL: https://github.com/apache/iceberg/issues/12042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] ExponentialHttpRequestRetryStrategy Enhancement [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on issue #12042: URL: https://github.com/apache/iceberg/issues/12042#issuecomment-2606783210 @AndyN5 Thanks for opening this issue, but we already have exponential backoff. You can find it here: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceber

Re: [I] Questions about the column iterator and format version. [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12047: URL: https://github.com/apache/iceberg/issues/12047#issuecomment-2607444838 We have some issues already marked for V2 Support for parquet files. That's really what we need to do to fix it. See https://github.com/apache/iceberg/issues/11371

Re: [I] NullPointerException when writing to Iceberg table using Spark 3.4.0 [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12037: URL: https://github.com/apache/iceberg/issues/12037#issuecomment-2607449743 As @nastra said without the full trace we can't really say anything about what went wrong. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] PyArrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #1555: URL: https://github.com/apache/iceberg-python/pull/1555#discussion_r1925454227 ## pyiceberg/partitioning.py: ## @@ -413,8 +413,10 @@ def partition_record_value(partition_field: PartitionField, value: Any, schema: the final partition rec

Re: [I] Default Iceberg Version in `EnvironmentContext` [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12035: URL: https://github.com/apache/iceberg/issues/12035#issuecomment-2607457215 Isn't this already the case? https://github.com/apache/iceberg/blob/98c58f7b9586b98b191cb36b88b33f857514b81b/core/src/main/java/org/apache/iceberg/EnvironmentContext.j

Re: [I] Default Iceberg Version in `EnvironmentContext` [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer closed issue #12035: Default Iceberg Version in `EnvironmentContext` URL: https://github.com/apache/iceberg/issues/12035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Issue with PositionDeletesTable in Apache Iceberg [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer closed issue #12031: Issue with PositionDeletesTable in Apache Iceberg URL: https://github.com/apache/iceberg/issues/12031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Issue with PositionDeletesTable in Apache Iceberg [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12031: URL: https://github.com/apache/iceberg/issues/12031#issuecomment-2607471563 I'm assuming this is an AI issue based on the formatting, lack of a real error or actual means to reproduce. I'm going to close and report this unless the @jjavieralonso has

Re: [I] Issue with PositionDeletesTable in Apache Iceberg [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12031: URL: https://github.com/apache/iceberg/issues/12031#issuecomment-2607473334 cc @Fokko also AI spam I think -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Add Support for AWS S3 Access Grants Integration [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer closed issue #12040: Add Support for AWS S3 Access Grants Integration URL: https://github.com/apache/iceberg/issues/12040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Add Support for AWS S3 Access Grants Integration [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on issue #12040: URL: https://github.com/apache/iceberg/issues/12040#issuecomment-2607483419 Another AI Issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925479680 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [PR] feat(puffin): Add PuffinReader [iceberg-rust]

2025-01-22 Thread via GitHub
waynexia commented on code in PR #892: URL: https://github.com/apache/iceberg-rust/pull/892#discussion_r1925473444 ## crates/iceberg/src/puffin/blob.rs: ## @@ -0,0 +1,38 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925479680 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925479680 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925479680 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [PR] Add data type/schema field/schema [iceberg-cpp]

2025-01-22 Thread via GitHub
lidavidm commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1925149114 ## src/iceberg/type.h: ## @@ -0,0 +1,397 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOT

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1925154433 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1230,6 +1265,22 @@ public Builder addSnapshot(Snapshot snapshot) { snapshotsById.put(snapsho

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1925162854 ## core/src/test/java/org/apache/iceberg/TestRowLineageMetadata.java: ## @@ -0,0 +1,326 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1925165978 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -312,6 +341,8 @@ public void testBackwardCompat() throws Exception { .isEqualTo(previou

Re: [PR] test: Introduce datafusion engine for executing sqllogictest. [iceberg-rust]

2025-01-22 Thread via GitHub
alamb commented on PR #895: URL: https://github.com/apache/iceberg-rust/pull/895#issuecomment-2607260468 I published datafusion-sqllogictest to crates.io (details here https://github.com/apache/datafusion/discussions/14229#discussioncomment-11916562) . Let us know if you have any problems!

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2025-01-22 Thread via GitHub
felixscherz commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2607599302 I will add that to the docs:) currently focusing on the moto side of things:) -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925106066 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -50,6 +54,32 @@ protected ParquetValueWriter createWriter(MessageType typ

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925113427 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -50,6 +54,32 @@ protected ParquetValueWriter createWriter(MessageType typ

Re: [I] feat: support metadata tables [iceberg-rust]

2025-01-22 Thread via GitHub
DeaconDesperado commented on issue #823: URL: https://github.com/apache/iceberg-rust/issues/823#issuecomment-2607288002 I'd like to attempt the FILES metadata table if not already being worked on. Thanks! -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Build: Nighly build for Iceberg REST fixtures [iceberg]

2025-01-22 Thread via GitHub
Fokko merged PR #12008: URL: https://github.com/apache/iceberg/pull/12008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Nighly build for Iceberg REST fixtures [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on PR #12008: URL: https://github.com/apache/iceberg/pull/12008#issuecomment-2607318917 Thanks @kevinjqliu šŸ™Œ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Improvement to unittest cases in catalog/registry.go [iceberg-go]

2025-01-22 Thread via GitHub
zeroshade commented on code in PR #264: URL: https://github.com/apache/iceberg-go/pull/264#discussion_r1925603606 ## catalog/registry.go: ## @@ -110,20 +110,20 @@ func GetRegisteredCatalogs() []string { // priority over any loaded config. // // If there is no "type" in the co

Re: [PR] feat(catalog): Make rest catalog auth scope configurable [iceberg-go]

2025-01-22 Thread via GitHub
zeroshade commented on code in PR #265: URL: https://github.com/apache/iceberg-go/pull/265#discussion_r1925619803 ## catalog/catalog.go: ## @@ -144,6 +150,7 @@ type options struct { sigv4Service string prefixstring authUri *url.

Re: [PR] feat(catalog): Make rest catalog auth scope configurable [iceberg-go]

2025-01-22 Thread via GitHub
zeroshade commented on code in PR #265: URL: https://github.com/apache/iceberg-go/pull/265#discussion_r1925623222 ## catalog/rest.go: ## @@ -491,11 +485,15 @@ func (r *RestCatalog) fetchAccessToken(cl *http.Client, creds string, opts *opti clientID, clientSecret

Re: [PR] Implement update for `remove-snapshots` action [iceberg-python]

2025-01-22 Thread via GitHub
grihabor commented on PR #1561: URL: https://github.com/apache/iceberg-python/pull/1561#issuecomment-2607712878 Sure! Thanks for the fast review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] ORC: Fix null map values and list elements in vectorized reads [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #12030: URL: https://github.com/apache/iceberg/pull/12030#discussion_r1925624176 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/GenericsHelpers.java: ## @@ -238,16 +254,17 @@ private static void assertEqualsUnsafe(Types.MapType map

Re: [I] UnsupportedOperationException: Unknown data file format during Spark query [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on issue #12033: URL: https://github.com/apache/iceberg/issues/12033#issuecomment-2606943881 Thanks for raising this @atharv9017. First I would recommend bumping Iceberg to a more recent version, 1.4.1 is pretty old already. Secondly, based on the information I can't debug t

Re: [I] UnsupportedOperationException: Unknown data file format during Spark query [iceberg]

2025-01-22 Thread via GitHub
Fokko closed issue #12033: UnsupportedOperationException: Unknown data file format during Spark query URL: https://github.com/apache/iceberg/issues/12033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Docs: Fix latest and nightly link on javadoc (according to site README.md) [iceberg]

2025-01-22 Thread via GitHub
jbonofre commented on PR #12023: URL: https://github.com/apache/iceberg/pull/12023#issuecomment-2607181878 I chatted with @Fokko and the javadoc version links to `latest` is a known issue and a bit complicated to fix. I propose to merge this PR (with `latest` & `nightly` links fix) an

Re: [PR] add iceberg_avro interface [iceberg-cpp]

2025-01-22 Thread via GitHub
wgtmac commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1925539978 ## cmake_modules/IcebergThirdpartyToolchain.cmake: ## @@ -126,3 +126,68 @@ endfunction() if(ICEBERG_ARROW) resolve_arrow_dependency() endif() + +# -

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
kevinjqliu commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925533289 ## .python-version: ## Review Comment: this is part of pyenv's local config, should not be checked into the repo ## poetry.lock: ##

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1925168044 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -60,6 +60,7 @@ private MetadataUpdateParser() {} static final String SET_PARTITION_STATIST

[I] Questions about the column iterator and format version. [iceberg]

2025-01-22 Thread via GitHub
sherman opened a new issue, #12047: URL: https://github.com/apache/iceberg/issues/12047 ### Query engine Spark 3.5.x Iceberg 1.7.0 ### Question According to the source code, the column iterator is created without specifying a version: ```java protected Primitive

Re: [PR] update PartitionSpec with snapshot'schema [iceberg]

2025-01-22 Thread via GitHub
mayankvadariya commented on PR #11196: URL: https://github.com/apache/iceberg/pull/11196#issuecomment-2607563392 @lurnagao-dahua thank you for creating a PR, are there any plans to re-open this PR? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2025-01-22 Thread via GitHub
kevinjqliu commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2607566104 I think that snippet would be great as part of the docs for S3 Tables :) -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] feat(puffin): Add PuffinReader [iceberg-rust]

2025-01-22 Thread via GitHub
liurenjie1024 commented on code in PR #892: URL: https://github.com/apache/iceberg-rust/pull/892#discussion_r1925017095 ## crates/iceberg/src/puffin/reader.rs: ## @@ -0,0 +1,126 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1924915138 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +87,34 @@ private boolean shouldRewrite(List group) { return enoughInput

[PR] Spark: Disable rewriting position deletes for V3 tables [iceberg]

2025-01-22 Thread via GitHub
nastra opened a new pull request, #12048: URL: https://github.com/apache/iceberg/pull/12048 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925494932 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar merged PR #12026: URL: https://github.com/apache/iceberg/pull/12026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925494932 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcT

Re: [I] Support Snapshot Expiration Operation [iceberg-python]

2025-01-22 Thread via GitHub
pp-gborodin commented on issue #516: URL: https://github.com/apache/iceberg-python/issues/516#issuecomment-2607530716 Hey @ndrluis, what's the status here? Do you have the document ready to be shared? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] # Title Feature Request / Improvement: Hardening Lock Mechanism for Retry and Interrupt Handling [iceberg]

2025-01-22 Thread via GitHub
Fokko closed issue #12034: # Title Feature Request / Improvement: Hardening Lock Mechanism for Retry and Interrupt Handling URL: https://github.com/apache/iceberg/issues/12034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add data type/schema field/schema [iceberg-cpp]

2025-01-22 Thread via GitHub
wgtmac commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1924967533 ## src/iceberg/type.h: ## @@ -0,0 +1,397 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTIC

Re: [PR] Add data type/schema field/schema [iceberg-cpp]

2025-01-22 Thread via GitHub
wgtmac commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1924965681 ## src/iceberg/type.h: ## @@ -0,0 +1,397 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTIC

Re: [PR] API: Add `UnknownType` [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #12012: URL: https://github.com/apache/iceberg/pull/12012#discussion_r1924964930 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -93,6 +95,10 @@ public SerializableFunction bind(Type type) { @Override public boolean c

Re: [PR] Core, Test: Parsing and Writing Tests for V3 Metadata [iceberg]

2025-01-22 Thread via GitHub
nastra commented on code in PR #12025: URL: https://github.com/apache/iceberg/pull/12025#discussion_r1925170994 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -314,8 +319,11 @@ public void testBackwardCompat() throws Exception { assertThat(metadata.s

Re: [I] NullPointerException when writing to Iceberg table using Spark 3.4.0 [iceberg]

2025-01-22 Thread via GitHub
nastra commented on issue #12037: URL: https://github.com/apache/iceberg/issues/12037#issuecomment-2607013932 @Bhhsaurabh can you please add some additional details like the actual full stack trace so that it's clear where the error is coming from? -- This is an automated message from the

Re: [PR] Spark: Fix reading 2 level array issue [iceberg]

2025-01-22 Thread via GitHub
nastra commented on PR #9515: URL: https://github.com/apache/iceberg/pull/9515#issuecomment-2607389264 @wallacms probably the best would be if you could bring up this issue on the [DEV mailing list](https://iceberg.apache.org/community/#mailing-lists) to reach the broader community -- Th

Re: [PR] API: Add `UnknownType` [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #12012: URL: https://github.com/apache/iceberg/pull/12012#discussion_r1924994313 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -431,6 +431,20 @@ public void testVariantUnsupported() { assertThat(bucket.canTransfor

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-22 Thread via GitHub
liurenjie1024 merged PR #882: URL: https://github.com/apache/iceberg-rust/pull/882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core, Test: Parsing and Writing Tests for V3 Metadata [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #12025: URL: https://github.com/apache/iceberg/pull/12025#discussion_r1925009567 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -314,8 +319,11 @@ public void testBackwardCompat() throws Exception { assertThat(metadata.sn

Re: [I] Template Name [iceberg]

2025-01-22 Thread via GitHub
Fokko closed issue #12038: Template Name URL: https://github.com/apache/iceberg/issues/12038 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsu

Re: [PR] test: Introduce datafusion engine for executing sqllogictest. [iceberg-rust]

2025-01-22 Thread via GitHub
liurenjie1024 commented on PR #895: URL: https://github.com/apache/iceberg-rust/pull/895#issuecomment-2606814519 > > Is it possible to take [datafusion_sqllogictest](https://github.com/apache/datafusion/blob/e9a77e0ea3e30b7f2718c9cea1fed023dca1f646/datafusion/sqllogictest/Car

Re: [PR] ORC: Fail when initial default support is required [iceberg]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #12026: URL: https://github.com/apache/iceberg/pull/12026#discussion_r1925143182 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -326,13 +327,20 @@ private static TypeDescription buildOrcProjection( orcType = origi

Re: [PR] Add data type/schema field/schema [iceberg-cpp]

2025-01-22 Thread via GitHub
lidavidm commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1925143148 ## src/iceberg/type.h: ## @@ -0,0 +1,397 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOT

Re: [PR] Spark: Fix reading 2 level array issue [iceberg]

2025-01-22 Thread via GitHub
wallacms commented on PR #9515: URL: https://github.com/apache/iceberg/pull/9515#issuecomment-2607355455 Can we re-open this (and merge it?) I've tested this code and it does fix the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Support Snapshot Expiration Operation [iceberg-python]

2025-01-22 Thread via GitHub
ndrluis commented on issue #516: URL: https://github.com/apache/iceberg-python/issues/516#issuecomment-2607732891 Hello @pp-gborodin. I have started the development by adding the necessary components before implementing the snapshot expiration operation (like #1285). This week, I will devel

Re: [PR] Flink: Add null check to writers to prevent resurrecting null values [iceberg]

2025-01-22 Thread via GitHub
mxm commented on code in PR #12049: URL: https://github.com/apache/iceberg/pull/12049#discussion_r1925639858 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetWriters.java: ## @@ -498,6 +498,12 @@ private static class RowDataWriter extends ParquetValu

Re: [I] Deprecate `snapshot-id` of `SetStatisticsUpdate` [iceberg-python]

2025-01-22 Thread via GitHub
ndrluis commented on issue #1556: URL: https://github.com/apache/iceberg-python/issues/1556#issuecomment-2607751865 @kevinjqliu Since we haven't released a version with snapshot_id yet, could we just remove it? What do you think? -- This is an automated message from the Apache Git Servic

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
gayatrikate04 commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925649913 ## .python-version: ## Review Comment: Got it! I will remove .python-version from the PR. ## mkdocs/docs/SUMMARY.md: ## @@ -30,7 +3

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
gayatrikate04 commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925651163 ## poetry.lock: ## Review Comment: Thank you for the suggestion! Iā€™m not very familiar with rebasing yet, but Iā€™m eager to learn. Could you guide me

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
gayatrikate04 commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925651563 ## mkdocs/mkdocs.yml: ## @@ -31,7 +31,8 @@ plugins: - mkdocstrings: handlers: python: - paths: [..] + paths: +

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
gayatrikate04 commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925651836 ## pyiceberg/table/inspect.py: ## @@ -57,7 +87,21 @@ def _get_snapshot(self, snapshot_id: Optional[int] = None) -> Snapshot: raise ValueError

Re: [PR] 1191:Added docstrings to the pyiceberg/table/inspect.py file [iceberg-python]

2025-01-22 Thread via GitHub
gayatrikate04 commented on code in PR #1533: URL: https://github.com/apache/iceberg-python/pull/1533#discussion_r1925652416 ## pyiceberg/table/inspect.py: ## @@ -95,7 +139,21 @@ def snapshots(self) -> "pa.Table": ) def entries(self, snapshot_id: Optional[int] = N

Re: [PR] REST: Add missing default HEAD endpoints and V1_COMMIT_TRANSACTION [iceberg]

2025-01-22 Thread via GitHub
nastra merged PR #11980: URL: https://github.com/apache/iceberg/pull/11980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925683533 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java: ## @@ -76,6 +80,46 @@ protected ParquetValueReader createReader( protected a

Re: [PR] Spark 3.5: Fix broadcasting specs in RewriteTablePath [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11982: URL: https://github.com/apache/iceberg/pull/11982#discussion_r1925676836 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -728,4 +724,22 @@ private String getMetadataLocation(Table t

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-22 Thread via GitHub
RussellSpitzer commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1925675533 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -60,6 +60,7 @@ private MetadataUpdateParser() {} static final String SET_PARTITION

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925697904 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -192,13 +222,17 @@ public Optional> visit( @Override public Opt

Re: [PR] ORC: Fix null map values and list elements in vectorized reads [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #12030: URL: https://github.com/apache/iceberg/pull/12030#discussion_r1925624176 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/GenericsHelpers.java: ## @@ -238,16 +254,17 @@ private static void assertEqualsUnsafe(Types.MapType map

Re: [PR] Remove slf4j-api reference in LICENSE as it's now excluded from the distributed jar files [iceberg]

2025-01-22 Thread via GitHub
jbonofre commented on PR #12052: URL: https://github.com/apache/iceberg/pull/12052#issuecomment-2608134014 We are talking about binary jar, so if we don't bundle the dependency in the jar, no need to mention it. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]

2025-01-22 Thread via GitHub
steveloughran commented on code in PR #10233: URL: https://github.com/apache/iceberg/pull/10233#discussion_r1925882790 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -166,23 +178,106 @@ public void deletePrefix(String prefix) { @Override public vo

Re: [PR] Flink 1.20: Support default values in Parquet reader [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11839: URL: https://github.com/apache/iceberg/pull/11839#discussion_r1925882243 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/RowDataUtil.java: ## @@ -63,6 +65,27 @@ public static Object convertConstant(Type type, Object value)

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925899152 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java: ## @@ -850,4 +919,42 @@ private TripleIterator firstNonNullColumn(List> columns) {

[PR] Refactor `truncate` transform types [iceberg-python]

2025-01-22 Thread via GitHub
Fokko opened a new pull request, #1562: URL: https://github.com/apache/iceberg-python/pull/1562 I think this aligns closer to the spec, and is also more friendly to the end-user: ![image](https://github.com/user-attachments/assets/1ae955a6-635f-4988-b964-fee471ebdad9) -- This

Re: [PR] PyArrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-22 Thread via GitHub
Fokko commented on code in PR #1555: URL: https://github.com/apache/iceberg-python/pull/1555#discussion_r1925902812 ## pyiceberg/partitioning.py: ## @@ -413,8 +413,10 @@ def partition_record_value(partition_field: PartitionField, value: Any, schema: the final partition rec

Re: [PR] ci(catalog): Improvement to unittest cases in catalog/registry.go [iceberg-go]

2025-01-22 Thread via GitHub
zeroshade merged PR #264: URL: https://github.com/apache/iceberg-go/pull/264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Remove slf4j-api reference in LICENSE as it's now excluded from the distributed jar files [iceberg]

2025-01-22 Thread via GitHub
jbonofre opened a new pull request, #12052: URL: https://github.com/apache/iceberg/pull/12052 https://github.com/apache/iceberg/pull/10665 updated the bundle jar files by removing slf4j-api. However, the `LICENSE` file in these bundle jar files has not been updated and still contain

Re: [PR] Remove slf4j-api reference in LICENSE as it's now excluded from the distributed jar files [iceberg]

2025-01-22 Thread via GitHub
jbonofre commented on PR #12052: URL: https://github.com/apache/iceberg/pull/12052#issuecomment-2607983153 @Fokko as discussed together šŸ˜„ @rdblue I'm doing a complete pass on the distributed artifacts to verify we are clean. This PR is the first one related to my findings. -- This

Re: [I] Deprecate `snapshot-id` of `SetStatisticsUpdate` [iceberg-python]

2025-01-22 Thread via GitHub
ndrluis commented on issue #1556: URL: https://github.com/apache/iceberg-python/issues/1556#issuecomment-2607984834 ok I can help with this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] feat(catalog): Make rest catalog auth scope configurable [iceberg-go]

2025-01-22 Thread via GitHub
zeroshade commented on code in PR #265: URL: https://github.com/apache/iceberg-go/pull/265#discussion_r1925801040 ## catalog/catalog.go: ## @@ -144,6 +150,7 @@ type options struct { sigv4Service string prefixstring authUri *url.

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925901471 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java: ## @@ -76,6 +70,46 @@ protected ParquetValueReader createReader( protected abstrac

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925911079 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java: ## @@ -63,6 +69,23 @@ public static ParquetValueReader position() { return new Posit

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912073 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912623 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -190,33 +210,49 @@ public Optional> visit( } @Override -public

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912623 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -190,33 +210,49 @@ public Optional> visit( } @Override -public

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925905904 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -50,6 +46,31 @@ protected ParquetValueWriter createWriter(MessageType type) {

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925905904 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -50,6 +46,31 @@ protected ParquetValueWriter createWriter(MessageType type) {

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-22 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925907570 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java: ## @@ -190,33 +210,49 @@ public Optional> visit( } @Override -public

[PR] feat: nan_value_counts support [iceberg-rust]

2025-01-22 Thread via GitHub
feniljain opened a new pull request, #907: URL: https://github.com/apache/iceberg-rust/pull/907 ## Issue Fixes #417 ## Description - We compute upper and lower bounds by relying on parquet statistics, but those statistics don't provide `nan_value_count`, so we have to i

  1   2   3   >