Re: [I] Better error messages when creating a table with unsupported types [iceberg-python]

2025-01-07 Thread via GitHub
DevChrisCross commented on issue #860: URL: https://github.com/apache/iceberg-python/issues/860#issuecomment-2576897755 @kevinjqliu @Fokko could you kindly check the PR please? Thank you! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] Modified exception objects being thrown when converting Pyarrow tables [iceberg-python]

2025-01-07 Thread via GitHub
DevChrisCross opened a new pull request, #1498: URL: https://github.com/apache/iceberg-python/pull/1498 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] [Discussion] Does iceberg-cpp require access to HDFS storage? [iceberg-cpp]

2025-01-07 Thread via GitHub
wgtmac commented on issue #21: URL: https://github.com/apache/iceberg-cpp/issues/21#issuecomment-2576866743 I have an open PR to introduce a `libiceberg_arrow` library which depends on Apache Arrow C++ to leverage its FileSystem implementation: https://github.com/apache/iceberg-cpp/pull/6.

Re: [I] [Discussion] Does iceberg-cpp require access to HDFS storage? [iceberg-cpp]

2025-01-07 Thread via GitHub
MisterRaindrop commented on issue #21: URL: https://github.com/apache/iceberg-cpp/issues/21#issuecomment-2576849433 Because I see that there is a storage implementation on the Java end. I feel that without an underlying file IO implementation, it’s unclear how effective HDFS would be. Of c

Re: [I] How do I know that the bloom filter configuration is successful? [iceberg]

2025-01-07 Thread via GitHub
hashmapybx commented on issue #11918: URL: https://github.com/apache/iceberg/issues/11918#issuecomment-2576835377 I'll try it also.the bloom filter is success. https://github.com/user-attachments/assets/98bf8ab3-81b9-4731-acae-75abba455f2f"; /> -- This is an automated messa

Re: [I] [Discussion] Does iceberg-cpp require access to HDFS storage? [iceberg-cpp]

2025-01-07 Thread via GitHub
wgtmac commented on issue #21: URL: https://github.com/apache/iceberg-cpp/issues/21#issuecomment-2576825868 No, I don't think we will add any storage as a requirement if we have a good interface for file I/O. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Metadata table scans as streams [iceberg-rust]

2025-01-07 Thread via GitHub
Xuanwo merged PR #870: URL: https://github.com/apache/iceberg-rust/pull/870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] [pyiceberg_core] Expose `IcebergTableProvider` to python [iceberg-rust]

2025-01-07 Thread via GitHub
Xuanwo commented on issue #865: URL: https://github.com/apache/iceberg-rust/issues/865#issuecomment-2576807109 That's nice! Thank you @kevinjqliu for pushing forward on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Ignore partition fields that are dropped from the current-schema [iceberg]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #11868: URL: https://github.com/apache/iceberg/pull/11868#discussion_r1906484702 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,24 @@ public String toString() { } } + public static class UnknownType extends P

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906482958 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from

[I] Does iceberg-cpp require access to HDFS storage? [iceberg-cpp]

2025-01-07 Thread via GitHub
MisterRaindrop opened a new issue, #21: URL: https://github.com/apache/iceberg-cpp/issues/21 Is there an introduction to the iceberg-cpp project? Does the underlying storage access HDFS or AWS? If it’s about HDFS, I might do some help. -- This is an automated message from the Apache Git S

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906470991 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906470991 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906464859 ## catalog/registry.go: ## @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

[PR] UpdateSchema does not respect transaction abort [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu opened a new pull request, #1497: URL: https://github.com/apache/iceberg-python/pull/1497 Closes #1253 While looking at #1433, i noticed that `UpdateSchema` does not respect the transaction semantic, i.e. it does not abort the whole transaction when any update in the trans

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1906457621 ## mkdocs/docs/api.md: ## @@ -961,20 +963,21 @@ Renaming a field in an Iceberg table is simple: ```python with table.update_schema() as update: update.

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1906456170 ## mkdocs/docs/api.md: ## @@ -961,20 +963,21 @@ Renaming a field in an Iceberg table is simple: ```python with table.update_schema() as update: update.

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1906456170 ## mkdocs/docs/api.md: ## @@ -961,20 +963,21 @@ Renaming a field in an Iceberg table is simple: ```python with table.update_schema() as update: update.

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906454658 ## catalog/registry.go: ## @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTI

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906453868 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906450780 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,227 @@ +import pytest Review Comment: Since this the file's name starts with `integration_*` instead of `test

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906450780 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,227 @@ +import pytest Review Comment: Since this the file's name starts with "integration_*" instead of "test

Re: [PR] Add table statistics [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on PR #1285: URL: https://github.com/apache/iceberg-python/pull/1285#issuecomment-2576706997 @ndrluis do you mind resolving the merge conflict here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add `all_manifests` metadata table with tests [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on PR #1241: URL: https://github.com/apache/iceberg-python/pull/1241#issuecomment-2576706003 @soumya-ghosh do you mind resolving the conflict? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#issuecomment-2576704408 can you run `poetry lock --no-update` for CI? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906433485 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,227 @@ +import pytest Review Comment: we _can_ merge the code with just the integration tests marked, but

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906431949 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,227 @@ +import pytest Review Comment: cool! i'd also `@pytest.mark` for now since we dont want this test

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1906423215 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from

Re: [I] [pyiceberg_core] Expose `IcebergTableProvider` to python [iceberg-rust]

2025-01-07 Thread via GitHub
kevinjqliu commented on issue #865: URL: https://github.com/apache/iceberg-rust/issues/865#issuecomment-2576694587 Got an example working by building the latest datafusion* libraries locally. Requires [apache/datafusion#13937](https://github.com/apache/datafusion/pull/13937) and new ver

[I] Kafka Connect: How to connect to Hive Metastore with SSL [iceberg]

2025-01-07 Thread via GitHub
duc-dn opened a new issue, #11925: URL: https://github.com/apache/iceberg/issues/11925 ### Query engine _No response_ ### Question Hi team, I am using Kafka Connect to sink data as format iceberg with Hive catalog However, my hive-metastore service requires SSL to c

Re: [PR] [Docs] Update spark-getting-started docs page to make the example valid [iceberg]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #11923: URL: https://github.com/apache/iceberg/pull/11923#discussion_r1906206762 ## spark/v3.3/spark-runtime/src/integration/java/org/apache/iceberg/spark/SmokeTest.java: ## @@ -66,25 +64,25 @@ public void testGettingStarted() throws IOExceptio

Re: [PR] Spec: add variant type [iceberg]

2025-01-07 Thread via GitHub
aihuaxu commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1906185106 ## version.txt: ## Review Comment: Thanks for catching that. It was added by accident. :) -- This is an automated message from the Apache Git Service. To res

Re: [I] (AWS Lake Formation shared resources) Iceberg tables in AWS Glue catalog has a different root namespace than the original [iceberg-python]

2025-01-07 Thread via GitHub
github-actions[bot] closed issue #845: (AWS Lake Formation shared resources) Iceberg tables in AWS Glue catalog has a different root namespace than the original URL: https://github.com/apache/iceberg-python/issues/845 -- This is an automated message from the Apache Git Service. To respond to

Re: [I] (AWS Lake Formation shared resources) Iceberg tables in AWS Glue catalog has a different root namespace than the original [iceberg-python]

2025-01-07 Thread via GitHub
github-actions[bot] commented on issue #845: URL: https://github.com/apache/iceberg-python/issues/845#issuecomment-2576464863 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] Add missing error codes to REST spec [iceberg]

2025-01-07 Thread via GitHub
github-actions[bot] closed issue #10570: Add missing error codes to REST spec URL: https://github.com/apache/iceberg/issues/10570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Consolidate Error Definitions in REST Spec [iceberg]

2025-01-07 Thread via GitHub
github-actions[bot] commented on issue #10571: URL: https://github.com/apache/iceberg/issues/10571#issuecomment-2576462406 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Consolidate Error Definitions in REST Spec [iceberg]

2025-01-07 Thread via GitHub
github-actions[bot] closed issue #10571: Consolidate Error Definitions in REST Spec URL: https://github.com/apache/iceberg/issues/10571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Add missing error codes to REST spec [iceberg]

2025-01-07 Thread via GitHub
github-actions[bot] commented on issue #10570: URL: https://github.com/apache/iceberg/issues/10570#issuecomment-2576462376 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906174773 ## catalog/rest_test.go: ## @@ -114,6 +114,39 @@ func (r *RestCatalogSuite) TestToken200() { r.Equal(r.configVals.Get("warehouse"), "s3://some-bucket") }

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906160998 ## catalog/registry.go: ## @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1906173910 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +86,30 @@ private boolean shouldRewrite(List group) { return enough

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1906173178 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +86,30 @@ private boolean shouldRewrite(List group) { return enough

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1906173178 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +86,30 @@ private boolean shouldRewrite(List group) { return enough

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1906173178 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +86,30 @@ private boolean shouldRewrite(List group) { return enough

Re: [PR] Core, Spark: Rewrite data files with high delete ratio [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #11825: URL: https://github.com/apache/iceberg/pull/11825#discussion_r1906168731 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -84,13 +86,30 @@ private boolean shouldRewrite(List group) { return enough

[PR] Call For Proposals Banner.html [iceberg]

2025-01-07 Thread via GitHub
Nhyi-streamlit opened a new pull request, #11924: URL: https://github.com/apache/iceberg/pull/11924 Creating a banner for folks to see and engage on the "Call For Proposals" for Iceberg 2025 Summit -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.71.1 to 1.72.0 [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade merged PR #243: URL: https://github.com/apache/iceberg-go/pull/243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Implement column projection [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906149085 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

Re: [PR] Implement column projection [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906149085 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

[PR] feat(catalog): Standardize Catalog create table function [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade opened a new pull request, #245: URL: https://github.com/apache/iceberg-go/pull/245 Adding a CreateTable function to the `Catalog` interface, standardizing the implementation that was initially created by #146 so that it isn't specific to the REST catalog and can be implemented by

[PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2025-01-07 Thread via GitHub
amogh-jahagirdar opened a new pull request, #11525: URL: https://github.com/apache/iceberg/pull/11525 This is a follow up to https://github.com/apache/iceberg/pull/11273/files# Instead of broadcasting a map with absolute paths for data files and delete files to executors, we could sh

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906143562 ## .palantir/revapi.yml: ## @@ -1171,6 +1171,28 @@ acceptedBreaks: \ java.util.function.Function, org.apache.iceberg.io.CloseableIterable,\ \ java.u

Re: [PR] Implement column projection [iceberg-python]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906142055 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906141975 ## parquet/src/test/java/org/apache/iceberg/parquet/TestInternalWriter.java: ## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906140419 ## parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetWriter.java: ## @@ -38,6 +50,19 @@ protected StructWriter createStructWriter(List> wr retu

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906140132 ## parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetWriter.java: ## @@ -38,6 +50,19 @@ protected StructWriter createStructWriter(List> wr retu

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906139217 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906138972 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906137219 ## catalog/glue.go: ## @@ -54,6 +57,50 @@ var ( _ Catalog = (*GlueCatalog)(nil) ) +func init() { + Register("glue", RegistrarFunc(func(_ string, pro

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906136606 ## catalog/registry.go: ## @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTI

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

2025-01-07 Thread via GitHub
HonahX closed pull request #11730: Core: Parsing and Writing Tests for V3 Metadata URL: https://github.com/apache/iceberg/pull/11730 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [Docs] Update spark-getting-started docs page to make the example valid [iceberg]

2025-01-07 Thread via GitHub
nickdelnano commented on PR #11923: URL: https://github.com/apache/iceberg/pull/11923#issuecomment-2576387003 Hi @kevinjqliu - I saw that you're a committer and recently looked at this doc page in https://github.com/apache/iceberg/pull/11845. Could you review this PR? -- This is an autom

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906134768 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132860 ## parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java: ## @@ -92,4 +127,232 @@ protected void set(Record struct, int pos, Object value) {

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132346 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java: ## @@ -76,6 +64,16 @@ protected ParquetValueReader createReader( protected abstrac

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132097 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java: ## @@ -114,113 +112,6 @@ public ParquetValueReader struct( } } - private c

Re: [PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-07 Thread via GitHub
kevinjqliu commented on code in PR #244: URL: https://github.com/apache/iceberg-go/pull/244#discussion_r1906120334 ## catalog/registry.go: ## @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

[PR] Build: Bump mypy-boto3-glue from 1.35.87 to 1.35.93 [iceberg-python]

2025-01-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1496: URL: https://github.com/apache/iceberg-python/pull/1496 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.35.87 to 1.35.93. Release notes Sourced from https://github.com/youtype/mypy_boto3_builder/release

[PR] Build: Bump boto3 from 1.35.88 to 1.35.93 [iceberg-python]

2025-01-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1495: URL: https://github.com/apache/iceberg-python/pull/1495 Bumps [boto3](https://github.com/boto/boto3) from 1.35.88 to 1.35.93. Commits https://github.com/boto/boto3/commit/7e5990c694164f96125d1362ed26bfec978c9e01";>7e5990c Merge

Re: [PR] [Docs] Update spark-getting-started docs page to make the example valid [iceberg]

2025-01-07 Thread via GitHub
nickdelnano commented on code in PR #11923: URL: https://github.com/apache/iceberg/pull/11923#discussion_r1906121331 ## docs/docs/spark-getting-started.md: ## @@ -77,21 +77,24 @@ Once your table is created, insert data using [`INSERT INTO`](spark-writes.md#in ```sql INSERT

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906125908 ## parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java: ## @@ -359,10 +250,10 @@ public ParquetValueReader primitive( ColumnDescript

[PR] [Docs] Update spark-getting-started docs page to make the example valid [iceberg]

2025-01-07 Thread via GitHub
nickdelnano opened a new pull request, #11923: URL: https://github.com/apache/iceberg/pull/11923 The [Spark Getting Started docs page](https://iceberg.apache.org/docs/nightly/spark-getting-started/) has intro Spark examples but they reference tables and columns that do not exist in the exa

Re: [PR] Implement column projection [iceberg-python]

2025-01-07 Thread via GitHub
gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906110946 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [I] create_changelog_view returns no record when end-timestamp is missing [iceberg]

2025-01-07 Thread via GitHub
sfc-gh-ygu commented on issue #11922: URL: https://github.com/apache/iceberg/issues/11922#issuecomment-2576343783 I suspect this line, https://github.com/apache/iceberg/blob/3eec2fa36c4ef92e6abaf516eed4e8be338c0649/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuild

Re: [I] create_changelog_view returns no record when end-timestamp is missing [iceberg]

2025-01-07 Thread via GitHub
flyrain commented on issue #11922: URL: https://github.com/apache/iceberg/issues/11922#issuecomment-2576344681 I suspect this line, https://github.com/apache/iceberg/blob/3eec2fa36c4ef92e6abaf516eed4e8be338c0649/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906095465 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Iceberg/Comet integration POC [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1906083743 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchUtil.java: ## @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Iceberg/Comet integration POC [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1906083743 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchUtil.java: ## @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Iceberg/Comet integration POC [iceberg]

2025-01-07 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1906083743 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchUtil.java: ## @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906077444 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[I] create_changelog_view returns no record when end-timestamp is missing [iceberg]

2025-01-07 Thread via GitHub
vinitamaloo-asu opened a new issue, #11922: URL: https://github.com/apache/iceberg/issues/11922 ### Apache Iceberg version 1.7.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Add two records in the table one after the other. `+---

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906075455 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2025-01-07 Thread via GitHub
zeroshade merged PR #146: URL: https://github.com/apache/iceberg-go/pull/146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906072738 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906069631 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906068027 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906067488 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906066082 ## parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Support convert orc timestamptz [iceberg]

2025-01-07 Thread via GitHub
pravin1406 commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2576237229 @ming95 I was facing this original issue and went about solving in the way you did, but it didn't work out for me,it was giving incorrect timestamp (future +5:30 (IST)). I see

Re: [I] Migration of ORC Backed Timestamp Without Zone Tables yields Timestamp With Zone columns [iceberg]

2025-01-07 Thread via GitHub
pravin1406 commented on issue #2245: URL: https://github.com/apache/iceberg/issues/2245#issuecomment-2576216258 @atifiu @RussellSpitzer Hi We are facing the same issue, tried making some code changes to ORCschemautil to bypass the error but it gives the incorrect timestamp as output . How

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906016284 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java: ## @@ -401,6 +403,17 @@ public ByteBuffer read(ByteBuffer reuse) { } } + publi

Re: [PR] Parquet: Internal writer and reader [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11904: URL: https://github.com/apache/iceberg/pull/11904#discussion_r1905999633 ## .palantir/revapi.yml: ## @@ -1171,6 +1171,28 @@ acceptedBreaks: \ java.util.function.Function, org.apache.iceberg.io.CloseableIterable,\ \ java.u

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905995521 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905993934 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905993243 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905992426 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905991506 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905990532 ## core/src/test/java/org/apache/iceberg/avro/TestInternalWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905975033 ## core/src/main/java/org/apache/iceberg/avro/InternalWriter.java: ## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [PR] Avro: Add internal writer [iceberg]

2025-01-07 Thread via GitHub
rdblue commented on code in PR #11919: URL: https://github.com/apache/iceberg/pull/11919#discussion_r1905968427 ## core/src/main/java/org/apache/iceberg/avro/ValueWriters.java: ## @@ -484,4 +489,16 @@ protected Object get(IndexedRecord struct, int pos) { return struct.get

  1   2   3   >