Re: [PR] AWS: Fix Catalog URI within VendedCredentialsProvider [iceberg]

2025-03-31 Thread via GitHub
nastra commented on code in PR #12612: URL: https://github.com/apache/iceberg/pull/12612#discussion_r2022218219 ## aws/src/test/java/org/apache/iceberg/aws/AwsClientPropertiesTest.java: ## @@ -213,8 +227,12 @@ public void refreshCredentialsEndpointWithRelativePath() { .

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-03-31 Thread via GitHub
nastra closed pull request #12418: Use delimited column names in CreateChangelogViewProcedure URL: https://github.com/apache/iceberg/pull/12418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] AWS: Fix Catalog URI within VendedCredentialsProvider [iceberg]

2025-03-31 Thread via GitHub
nastra commented on code in PR #12612: URL: https://github.com/apache/iceberg/pull/12612#discussion_r2022217974 ## aws/src/test/java/org/apache/iceberg/aws/AwsClientPropertiesTest.java: ## @@ -213,8 +227,12 @@ public void refreshCredentialsEndpointWithRelativePath() { .

Re: [PR] AWS: Fix Catalog URI within VendedCredentialsProvider [iceberg]

2025-03-31 Thread via GitHub
nastra commented on code in PR #12612: URL: https://github.com/apache/iceberg/pull/12612#discussion_r2022217759 ## aws/src/test/java/org/apache/iceberg/aws/AwsClientPropertiesTest.java: ## @@ -161,8 +165,12 @@ public void refreshCredentialsEndpointWithOAuthToken() { .ex

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-03-31 Thread via GitHub
zhjwpku commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2022216996 ## src/iceberg/arrow/arrow_error_transform_internal.h: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor li

Re: [PR] AWS: Add parameter of excluding non-current fields in Glue [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on PR #12664: URL: https://github.com/apache/iceberg/pull/12664#issuecomment-2768177860 Displaying non-current columns is intentional in Glue, as users may use LakeFormation and need to access dropped columns. Users should not rely on Glue for the latest table status,

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-03-31 Thread via GitHub
zhjwpku commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2022210534 ## src/iceberg/arrow/io/CMakeLists.txt: ## @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on code in PR #12671: URL: https://github.com/apache/iceberg/pull/12671#discussion_r2022177712 ## aws/src/integration/java/org/apache/iceberg/aws/dynamodb/TestDynamoDbCatalog.java: ## @@ -56,8 +58,8 @@ import software.amazon.awssdk.services.s3.model.HeadOb

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on code in PR #12671: URL: https://github.com/apache/iceberg/pull/12671#discussion_r2022173736 ## docs/docs/aws.md: ## @@ -313,6 +313,18 @@ Here are the configurations that users can tune related to this feature: | s3.multipart.threshold| 1.5

Re: [PR] Doc: Remove Warning for resolved issue #11147. [iceberg]

2025-03-31 Thread via GitHub
slfan1989 commented on PR #12694: URL: https://github.com/apache/iceberg/pull/12694#issuecomment-2768099550 > Can we add reference to what PR addressed the issue? Thank you for your message! I have added the reference to the PR(#11157) that addresses the issue in the description.

Re: [PR] Docs: Add warning about `snapshot_ids` arg in `expired_snapshots` procedure [iceberg]

2025-03-31 Thread via GitHub
github-actions[bot] closed pull request #12291: Docs: Add warning about `snapshot_ids` arg in `expired_snapshots` procedure URL: https://github.com/apache/iceberg/pull/12291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Docs: Add warning about `snapshot_ids` arg in `expired_snapshots` procedure [iceberg]

2025-03-31 Thread via GitHub
github-actions[bot] commented on PR #12291: URL: https://github.com/apache/iceberg/pull/12291#issuecomment-2767700779 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Add partition field/partition spec [iceberg-cpp]

2025-03-31 Thread via GitHub
wgtmac commented on code in PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#discussion_r2022059225 ## src/iceberg/CMakeLists.txt: ## @@ -23,7 +23,11 @@ set(ICEBERG_SOURCES schema.cc schema_field.cc schema_internal.cc -type.cc) +type.cc Review C

Re: [I] Doc update - https://github.com/apache/iceberg/blob/main/docs/docs/aws.md#s3-tags [iceberg]

2025-03-31 Thread via GitHub
XBaith commented on issue #12680: URL: https://github.com/apache/iceberg/issues/12680#issuecomment-2767969472 Hi @opensource4me @manuzhang. I can handle this, can you assign this to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] CORE: Allow HTTPClient to parse headers from properties. [iceberg]

2025-03-31 Thread via GitHub
wolflex888 commented on code in PR #12595: URL: https://github.com/apache/iceberg/pull/12595#discussion_r2022056893 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -332,6 +343,46 @@ public void testInitializeWithBadArguments() throws IOException {

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-03-31 Thread via GitHub
Guosmilesmile commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2767901065 @abharath9 @pvary @stevenzwu Since this backport has been kept as a Draft with no response for a long time, can I open a new PR to continue and finish this backport? -- This is

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on PR #313: URL: https://github.com/apache/iceberg-go/pull/313#issuecomment-2767833676 > @xuhui-lu looks like the integration test still fails. otherwise things look pretty good to me. just need to get the tests fixed Not sure what happened on ci. Let me just verify

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-03-31 Thread via GitHub
nastra commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2021074800 ## core/src/main/java/org/apache/iceberg/MetadataUpdate.java: ## @@ -328,20 +328,20 @@ public void applyTo(TableMetadata.Builder metadataBuilder) { } } - c

Re: [I] Snapshot chain getting broken - data incorrectly removed [iceberg]

2025-03-31 Thread via GitHub
github-actions[bot] commented on issue #11243: URL: https://github.com/apache/iceberg/issues/11243#issuecomment-2767700634 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Doc update - https://github.com/apache/iceberg/blob/main/docs/docs/aws.md#s3-tags [iceberg]

2025-03-31 Thread via GitHub
manuzhang commented on issue #12680: URL: https://github.com/apache/iceberg/issues/12680#issuecomment-2767734788 @opensource4me would you mind submitting a PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Core: ability to inject an AuthManager in RESTCatalog [iceberg]

2025-03-31 Thread via GitHub
gh-yzou commented on PR #12655: URL: https://github.com/apache/iceberg/pull/12655#issuecomment-2767714302 I also want to add some more context for the Polaris use case, the kind of capability we want is not just provide a customized authentication manager but also share the same authenticat

Re: [PR] Set field-id when needed [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on code in PR #1867: URL: https://github.com/apache/iceberg-python/pull/1867#discussion_r2021942387 ## pyiceberg/io/pyarrow.py: ## @@ -1777,7 +1777,7 @@ def struct( field_arrays.append(array) fields.append(self._construct_fi

Re: [I] Inconsistent id definition on Flink resolvedSchema conversion to iceberg schema [iceberg]

2025-03-31 Thread via GitHub
github-actions[bot] commented on issue #11128: URL: https://github.com/apache/iceberg/issues/11128#issuecomment-2767700585 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Does main branch reference reset requiring a clean up of snapshot logs [iceberg]

2025-03-31 Thread via GitHub
github-actions[bot] commented on issue #11109: URL: https://github.com/apache/iceberg/issues/11109#issuecomment-2767700561 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] allow dashes in glue database and table names [iceberg]

2025-03-31 Thread via GitHub
jrouly commented on PR #12677: URL: https://github.com/apache/iceberg/pull/12677#issuecomment-2767695771 @xiaoxuandev It's downstream validation, then, not Glue validation? Doesn't really make sense why it would be considered specifically Glue validation in that case... -- This is an aut

Re: [PR] allow dashes in glue database and table names [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on PR #12677: URL: https://github.com/apache/iceberg/pull/12677#issuecomment-2767690604 > @xiaoxuandev the issue, in my mind at least, is less that glue validation is on or off by default and more that it is validating incorrectly. Legal glue table and database names m

Re: [PR] fix: Preserve list name in schema conversion [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n commented on PR #1149: URL: https://github.com/apache/iceberg-rust/pull/1149#issuecomment-2767669247 Would like to request some help here. I believe the catalog is running into an error while deserializing the response to query the catalog. Here is the message: ```threa

Re: [PR] API, Core: Geospatial bounds and spatial predicates [iceberg]

2025-03-31 Thread via GitHub
jiayuasu commented on PR #12667: URL: https://github.com/apache/iceberg/pull/12667#issuecomment-2767657136 @szehon-ho @rdblue Can you review when you have time? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] doc: Clarify `arrow_schema_to_schema` requires fields with field id [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n commented on PR #1151: URL: https://github.com/apache/iceberg-rust/pull/1151#issuecomment-2767606499 This should be the last of the prs for the 0.5.0 milestone. I think we should introduce some new `good first issues` for newcomers of the codebase though since these prs have rem

[PR] doc: Clarify `arrow_schema_to_schema` requires fields with field id [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n opened a new pull request, #1151: URL: https://github.com/apache/iceberg-rust/pull/1151 ## Which issue does this PR close? - Closes #1061 . ## What changes are included in this PR? Add doc for arrow_schema_to_schema ## Are these changes tested?

[PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n opened a new pull request, #1150: URL: https://github.com/apache/iceberg-rust/pull/1150 ## Which issue does this PR close? - Closes #timestamp. ## What changes are included in this PR? Add bucket transforms for `TimestampNs` and TimestampTzNs` #

[PR] Build: Bump griffe from 1.6.3 to 1.7.1 [iceberg-python]

2025-03-31 Thread via GitHub
dependabot[bot] opened a new pull request, #1873: URL: https://github.com/apache/iceberg-python/pull/1873 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 1.6.3 to 1.7.1. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

[PR] Build: Bump mkdocs-material from 9.6.9 to 9.6.10 [iceberg-python]

2025-03-31 Thread via GitHub
dependabot[bot] opened a new pull request, #1875: URL: https://github.com/apache/iceberg-python/pull/1875 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.9 to 9.6.10. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>m

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-03-31 Thread via GitHub
lliangyu-lin commented on PR #12671: URL: https://github.com/apache/iceberg/pull/12671#issuecomment-2767573003 cc: @nastra @xiaoxuandev I made some updates based on the reviews. Feel free to take another look when you have time. -- This is an automated message from the Apache Git Service.

[PR] Build: Bump datafusion from 45.2.0 to 46.0.0 [iceberg-python]

2025-03-31 Thread via GitHub
dependabot[bot] opened a new pull request, #1872: URL: https://github.com/apache/iceberg-python/pull/1872 Bumps [datafusion](https://github.com/apache/datafusion-python) from 45.2.0 to 46.0.0. Commits https://github.com/apache/datafusion-python/commit/1aa9cb98bc2cc070048027ac96

[PR] Build: Bump mkdocstrings from 0.29.0 to 0.29.1 [iceberg-python]

2025-03-31 Thread via GitHub
dependabot[bot] opened a new pull request, #1871: URL: https://github.com/apache/iceberg-python/pull/1871 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.29.0 to 0.29.1. Release notes Sourced from https://github.com/mkdocstrings/mkdocstrings/releases";>mkd

[PR] Build: Bump coverage from 7.7.1 to 7.8.0 [iceberg-python]

2025-03-31 Thread via GitHub
dependabot[bot] opened a new pull request, #1870: URL: https://github.com/apache/iceberg-python/pull/1870 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.7.1 to 7.8.0. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's cha

Re: [PR] Flink: Backport support source watermark for flink sql windows [iceberg]

2025-03-31 Thread via GitHub
swapna267 commented on PR #12697: URL: https://github.com/apache/iceberg/pull/12697#issuecomment-2767553905 @pvary this is a clean backport. There were some checks not related to my code were failing in first commit, so had to make dummy commit to re-trigger tests. -- This is an automate

Re: [PR] fix: Preserve list name in schema conversion [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n commented on code in PR #1149: URL: https://github.com/apache/iceberg-rust/pull/1149#discussion_r2021852830 ## crates/iceberg/src/spec/datatypes.rs: ## @@ -633,11 +633,11 @@ impl NestedField { } /// Construct list type's element field. -pub fn list_el

[PR] fix: Preserve list name in schema conversion [iceberg-rust]

2025-03-31 Thread via GitHub
jonathanc-n opened a new pull request, #1149: URL: https://github.com/apache/iceberg-rust/pull/1149 ## Which issue does this PR close? - part of #1039 . ## What changes are included in this PR? Preserve list name in serialization and deserialization. ## Ar

Re: [PR] Add partition field/partition spec [iceberg-cpp]

2025-03-31 Thread via GitHub
wgtmac commented on code in PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#discussion_r2020465204 ## src/iceberg/partition_spec.h: ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-03-31 Thread via GitHub
zratkai commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2020865143 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java: ## @@ -0,0 +1,264 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
mrcnc commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021788810 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +//

Re: [PR] spec: Variant lower/upper bounds [iceberg]

2025-03-31 Thread via GitHub
aihuaxu commented on code in PR #12658: URL: https://github.com/apache/iceberg/pull/12658#discussion_r2021739556 ## format/spec.md: ## @@ -1558,6 +1561,7 @@ The binary single-value serialization can be used to store the lower and upper b |--|--

Re: [I] Upsert with NULL value not working [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on issue #1835: URL: https://github.com/apache/iceberg-python/issues/1835#issuecomment-2766916088 Thanks for reporting the issue @Benjamin-Lemaire #1861 should resolve this and will be included in the upcoming 0.9.1 release -- This is an automated message from the Ap

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
zeroshade commented on PR #313: URL: https://github.com/apache/iceberg-go/pull/313#issuecomment-2767423047 @xuhui-lu looks like the integration test still fails. otherwise things look pretty good to me. just need to get the tests fixed -- This is an automated message from the Apache Git S

Re: [I] Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on issue #1864: URL: https://github.com/apache/iceberg-python/issues/1864#issuecomment-2767367406 🤔 Thanks for letting us know 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-03-31 Thread via GitHub
zeroshade opened a new pull request, #369: URL: https://github.com/apache/iceberg-go/pull/369 This creates the initial implementation to write a stream of data or a materialized Table of arrow data to unpartitioned tables. Implementing writing with partitions will be done in a follow-up PR

Re: [PR] AWS: Fix NullPointerException for glue tables without storage descriptor [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on code in PR #12551: URL: https://github.com/apache/iceberg/pull/12551#discussion_r2021738899 ## aws/src/test/java/org/apache/iceberg/aws/glue/TestIcebergToGlueConverter.java: ## @@ -384,4 +384,71 @@ public void testSetTableInputInformationWithExistingTab

Re: [PR] Set field-id when needed [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #1867: URL: https://github.com/apache/iceberg-python/pull/1867#discussion_r2021735921 ## pyiceberg/io/pyarrow.py: ## @@ -1777,7 +1777,7 @@ def struct( field_arrays.append(array) fields.append(self._construct_field(f

Re: [PR] Pyarrow data type, default to small type and fix large type override [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #1859: URL: https://github.com/apache/iceberg-python/pull/1859#discussion_r2021694553 ## mkdocs/docs/configuration.md: ## @@ -199,7 +199,7 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya | Key

Re: [PR] add scan tests with null values [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #1865: URL: https://github.com/apache/iceberg-python/pull/1865#discussion_r2021670570 ## tests/io/test_pyarrow.py: ## @@ -2317,3 +2321,66 @@ def test_pyarrow_io_multi_fs() -> None: # Same PyArrowFileIO instance resolves local file input t

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-03-31 Thread via GitHub
RussellSpitzer commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021528707 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3

Re: [I] Freshly created table instance is not write-able [iceberg-go]

2025-03-31 Thread via GitHub
zeroshade commented on issue #368: URL: https://github.com/apache/iceberg-go/issues/368#issuecomment-2767096599 create table has a `WithProperties` option [here](https://github.com/apache/iceberg-go/blob/main/catalog/catalog.go#L151) which should be sufficient for what you need, right? --

Re: [I] ensure `name-mapping` exists [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on issue #1866: URL: https://github.com/apache/iceberg-python/issues/1866#issuecomment-2767151373 We should not set the name-mapping when it isn't needed. Looking at the code, `ensureNameMappingPresent` is used when: - [Add files without field-IDs](https://github.com

Re: [I] Freshly created table instance is not write-able [iceberg-go]

2025-03-31 Thread via GitHub
laskoviymishka commented on issue #368: URL: https://github.com/apache/iceberg-go/issues/368#issuecomment-2767149787 Yes, missed that one, issue can be closed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Freshly created table instance is not write-able [iceberg-go]

2025-03-31 Thread via GitHub
laskoviymishka closed issue #368: Freshly created table instance is not write-able URL: https://github.com/apache/iceberg-go/issues/368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Core: Extended header support for RESTClient implementations [iceberg]

2025-03-31 Thread via GitHub
zratkai commented on code in PR #12194: URL: https://github.com/apache/iceberg/pull/12194#discussion_r2020947113 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -132,7 +132,7 @@ private HTTPClient( * shares the same base uri, mapper, and HTTP client as t

Re: [PR] AWS: Support StaticCredentialsProvider in DefaultAwsClientFactory [iceberg]

2025-03-31 Thread via GitHub
xiaoxuandev commented on PR #12695: URL: https://github.com/apache/iceberg/pull/12695#issuecomment-2767045401 > @xiaoxuandev The [default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) supports setting these values via envi

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-03-31 Thread via GitHub
flyrain commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2767130138 Looks like the magic words don't work. Can you submit an empty or trivial commit to retrigger it? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-03-31 Thread via GitHub
flyrain commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2767127261 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-03-31 Thread via GitHub
andyglow commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2767119764 something is not right with tests based off of `RESTServerExtension` ``` java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:45207 java.net.BindException: Address already in

[I] ensure `name-mapping` exists [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu opened a new issue, #1866: URL: https://github.com/apache/iceberg-python/issues/1866 ### Feature Request / Improvement Should we set `name-mapping` on table creation? Spark create table [ensures that name-mapping exists](https://github.com/apache/iceberg/blob/main/s

Re: [PR] Spec: update to reflect lineage is required [iceberg]

2025-03-31 Thread via GitHub
danielcweeks commented on code in PR #12580: URL: https://github.com/apache/iceberg/pull/12580#discussion_r2021566970 ## format/spec.md: ## @@ -408,16 +406,17 @@ When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its Values for `_row_id` and `_last_u

[I] Freshly created table instance is not write-able [iceberg-go]

2025-03-31 Thread via GitHub
laskoviymishka opened a new issue, #368: URL: https://github.com/apache/iceberg-go/issues/368 ### Apache Iceberg version None ### Please describe the bug 🐞 Table instance has an [IO](https://github.com/apache/iceberg-go/blob/main/table/table.go#L58) reference, which may

Re: [PR] AWS: Support StaticCredentialsProvider in DefaultAwsClientFactory [iceberg]

2025-03-31 Thread via GitHub
danielcweeks commented on PR #12695: URL: https://github.com/apache/iceberg/pull/12695#issuecomment-2766961012 @xiaoxuandev The d[efault credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) supports setting these values via envir

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#issuecomment-2766914867 thanks for the review @Fokko i opened #1865 to test the scan logic with nulls -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] add scan tests with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on code in PR #1865: URL: https://github.com/apache/iceberg-python/pull/1865#discussion_r2021466534 ## tests/io/test_pyarrow.py: ## @@ -2317,3 +2321,66 @@ def test_pyarrow_io_multi_fs() -> None: # Same PyArrowFileIO instance resolves local file in

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-03-31 Thread via GitHub
danielcweeks commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2021463331 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -1720,143 +1717,15 @@ public void testInvalidAPIUsage()

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021460617 ## io/io.go: ## @@ -258,6 +258,11 @@ func inferFileIOFromSchema(ctx context.Context, path string, props map[string]st bucket = memblob.OpenBucket(nil

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021457890 ## io/azure_integration_test.go: ## @@ -0,0 +1,148 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021457489 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +

Re: [PR] add scan tests with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on code in PR #1865: URL: https://github.com/apache/iceberg-python/pull/1865#discussion_r2021453036 ## tests/io/test_pyarrow.py: ## @@ -2317,3 +2321,66 @@ def test_pyarrow_io_multi_fs() -> None: # Same PyArrowFileIO instance resolves local file in

Re: [PR] add scan tests with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on code in PR #1865: URL: https://github.com/apache/iceberg-python/pull/1865#discussion_r2021452599 ## tests/io/test_pyarrow.py: ## @@ -2317,3 +2321,66 @@ def test_pyarrow_io_multi_fs() -> None: # Same PyArrowFileIO instance resolves local file in

[PR] add scan tests with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu opened a new pull request, #1865: URL: https://github.com/apache/iceberg-python/pull/1865 # Rationale for this change Test scan with null values Similar to https://github.com/apache/iceberg-rust/pull/1045 # Are these changes tested? # Ar

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu merged PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] Upsert with NULL value not working [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu closed issue #1835: Upsert with NULL value not working URL: https://github.com/apache/iceberg-python/issues/1835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[PR] Flink: Backport support source watermark for flink sql windows [iceberg]

2025-03-31 Thread via GitHub
swapna267 opened a new pull request, #12697: URL: https://github.com/apache/iceberg/pull/12697 Backport changes in https://github.com/apache/iceberg/pull/12191 to 1.18,1.19 Iceberg Source to support Source Watermark, so it can be used in Flink WINDOW functions. https://github.com/

Re: [I] Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible [iceberg-python]

2025-03-31 Thread via GitHub
carc-harsh closed issue #1864: Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible URL: https://github.com/apache/iceberg-python/issues/1864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible [iceberg-python]

2025-03-31 Thread via GitHub
carc-harsh commented on issue #1864: URL: https://github.com/apache/iceberg-python/issues/1864#issuecomment-2766840535 > Hmmm, `2025.3.1.*>=2023.1.0` holds, right? 🤔 Yes, I'm not sure what the issue is, but the `s3fs` package updated to `2025.3.2` and that solves the issue https:/

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2025-03-31 Thread via GitHub
XBaith commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2765466418 Hi @nastra , I’ve also encountered this issue with the Glue catalog. Specifically, commands like `SHOW DATABASES` or `SHOW TABLES` return an empty list because the interfaces refere

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
zeroshade commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021408261 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file

Re: [PR] Build: Bump datamodel-code-generator from 0.28.4 to 0.28.5 [iceberg]

2025-03-31 Thread via GitHub
Fokko merged PR #12683: URL: https://github.com/apache/iceberg/pull/12683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021399459 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021395175 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +

Re: [PR] Flink: backport support create table like in flink catalog [iceberg]

2025-03-31 Thread via GitHub
swapna267 commented on PR #12679: URL: https://github.com/apache/iceberg/pull/12679#issuecomment-2766798324 Thanks @pvary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on issue #1864: URL: https://github.com/apache/iceberg-python/issues/1864#issuecomment-2766785127 Hmmm, `2025.3.1.*>=2023.1.0` holds, right? 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-03-31 Thread via GitHub
flyrain commented on code in PR #12418: URL: https://github.com/apache/iceberg/pull/12418#discussion_r2021374263 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/CreateChangelogViewProcedure.java: ## @@ -210,7 +210,12 @@ private Dataset removeCarryoverRows(D

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021373556 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-03-31 Thread via GitHub
zeroshade commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2021316142 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-03-31 Thread via GitHub
RussellSpitzer commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021369062 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#discussion_r2021358767 ## tests/table/test_upsert.py: ## @@ -509,3 +509,39 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: ValueError, match="Join colum

[I] Installing "pyiceberg[hive,s3fs]==0.9.0" causes pip ResolutionImpossible [iceberg-python]

2025-03-31 Thread via GitHub
carc-harsh opened a new issue, #1864: URL: https://github.com/apache/iceberg-python/issues/1864 ### Apache Iceberg version 0.9.0 (latest release) ### Please describe the bug 🐞 Logs: ``` ERROR: Cannot install pyiceberg and pyiceberg[hive,s3fs]==0.9.0 because these p

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#issuecomment-2766747183 now im wondering if nulls are properly handled when we convert iceberg expressions to pyarrow expressions https://github.com/apache/iceberg-python/blob/1a5e32ab234ed180b4

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021334130 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2021334130 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
Fokko commented on code in PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#discussion_r2021344261 ## tests/table/test_upsert.py: ## @@ -509,3 +509,39 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: ValueError, match="Join colum

[PR] Support quoted column identifiers for scan `row_filter` string argument [iceberg-python]

2025-03-31 Thread via GitHub
norton120 opened a new pull request, #1863: URL: https://github.com/apache/iceberg-python/pull/1863 # Rationale for this change Our data lake uses old-school Kimball style quoted column names ("User ID", "Customer Name" etc). The string parser for `row_filter` was unable to parse this. N

Re: [PR] fix `upsert` with null values [iceberg-python]

2025-03-31 Thread via GitHub
kevinjqliu commented on code in PR #1861: URL: https://github.com/apache/iceberg-python/pull/1861#discussion_r2021305248 ## tests/table/test_upsert.py: ## @@ -509,3 +509,39 @@ def test_upsert_without_identifier_fields(catalog: Catalog) -> None: ValueError, match="Join

[PR] Documented `row_filter` expressions [iceberg-python]

2025-03-31 Thread via GitHub
norton120 opened a new pull request, #1862: URL: https://github.com/apache/iceberg-python/pull/1862 # Rationale for this change scan's `row_filter` param is not super intuitive. I got tired of reading over the expression and parser code as I'm trying to build out statements, so I had som

  1   2   >