Re: [PR] Docs: Add versioned docs for 1.8.1 [iceberg]

2025-02-27 Thread via GitHub
nastra merged PR #12411: URL: https://github.com/apache/iceberg/pull/12411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Docs: Remove Hive runtime jar link from latest release [iceberg]

2025-02-27 Thread via GitHub
nastra merged PR #12422: URL: https://github.com/apache/iceberg/pull/12422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] fix: upgrade spark version [iceberg-rust]

2025-02-27 Thread via GitHub
ZENOTME commented on PR #1015: URL: https://github.com/apache/iceberg-rust/pull/1015#issuecomment-2689953572 cc @kevinjqliu @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] fix: upgrade spark version [iceberg-rust]

2025-02-27 Thread via GitHub
ZENOTME opened a new pull request, #1015: URL: https://github.com/apache/iceberg-rust/pull/1015 ## Which issue does this PR close? Spark 3.5.4 seems to have disappeared from [here](https://dlcdn.apache.org/spark/), there's 3.5.5 instead, so upgrading to that -

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
nastra commented on PR #12401: URL: https://github.com/apache/iceberg/pull/12401#issuecomment-2689940403 @amogh-jahagirdar could you also take a look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] tbl.append(df): schema validation of tbl & df during compares the order & data types [iceberg-python]

2025-02-27 Thread via GitHub
sivaraman-ai commented on issue #1088: URL: https://github.com/apache/iceberg-python/issues/1088#issuecomment-2689937834 this is resolved, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat: refine record batch project [iceberg-rust]

2025-02-27 Thread via GitHub
ZENOTME commented on PR #1014: URL: https://github.com/apache/iceberg-rust/pull/1014#issuecomment-2689907474 Potential improvement: For now, we will trave the whole record in visitor pattern, but in some case we don't need to travese child column. E.g. the map column is passthrough, we don'

Re: [PR] feat: refine record batch project [iceberg-rust]

2025-02-27 Thread via GitHub
ZENOTME commented on PR #1014: URL: https://github.com/apache/iceberg-rust/pull/1014#issuecomment-2689902531 cc @liurenjie1024 @sdd @Xuanwo @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Remove workaround for TimestampNTZType in TestHelpers [iceberg]

2025-02-27 Thread via GitHub
nastra closed issue #11945: Remove workaround for TimestampNTZType in TestHelpers URL: https://github.com/apache/iceberg/issues/11945 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Core,Api: Add overwrite option when register external table to catalog [iceberg]

2025-02-27 Thread via GitHub
stevenzwu commented on code in PR #12228: URL: https://github.com/apache/iceberg/pull/12228#discussion_r1972654788 ## core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java: ## @@ -71,23 +70,35 @@ public Table loadTable(TableIdentifier identifier) { } @Override

Re: [PR] Core,Api: Add overwrite option when register external table to catalog [iceberg]

2025-02-27 Thread via GitHub
stevenzwu commented on code in PR #12228: URL: https://github.com/apache/iceberg/pull/12228#discussion_r1972654788 ## core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java: ## @@ -71,23 +70,35 @@ public Table loadTable(TableIdentifier identifier) { } @Override

Re: [PR] Docs: Describe how to handle versioned docs/javadoc during a release [iceberg]

2025-02-27 Thread via GitHub
amogh-jahagirdar merged PR #12413: URL: https://github.com/apache/iceberg/pull/12413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] feat: make output data path of table with identity timestamp partition consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu commented on PR #1736: URL: https://github.com/apache/iceberg-python/pull/1736#issuecomment-2689629069 > Hi @sharkdtu thank you for working on this PR! 😊 > > I think consistency is great, but Iceberg currently does not require that we guarantee consistent paths (unlike Hive s

[PR] Docs: Remove Hive runtime jar link from 1.8.0 release [iceberg]

2025-02-27 Thread via GitHub
manuzhang opened a new pull request, #12422: URL: https://github.com/apache/iceberg/pull/12422 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] OpenAPI: Use more clear language in recommending error responses [iceberg]

2025-02-27 Thread via GitHub
sungwy commented on code in PR #12376: URL: https://github.com/apache/iceberg/pull/12376#discussion_r1974643193 ## open-api/rest-catalog-open-api.yaml: ## @@ -4566,7 +4568,9 @@ components: AuthenticationTimeoutResponse: description: -Credentials have timed

Re: [I] Partition info is not displayed in job description for RewritePositionDeleteFilesSparkAction [iceberg]

2025-02-27 Thread via GitHub
manuzhang commented on issue #12414: URL: https://github.com/apache/iceberg/issues/12414#issuecomment-2689598283 @szehon-ho do you have an example? The UT passed without partition coercion, and I've not seen any issues from `RewriteDataFilesSparkAction` without it. -- This is an automated

Re: [I] The identity partition path of timestamp type is inconsistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu commented on issue #1735: URL: https://github.com/apache/iceberg-python/issues/1735#issuecomment-2689616876 > Hi [@sharkdtu](https://github.com/sharkdtu) my understanding is that Iceberg does not make any guarantees on the paths of the data files, as it relies on links to connect

[PR] Fix: REST Catalog should retry on 401 status code [iceberg-python]

2025-02-27 Thread via GitHub
sungwy opened a new pull request, #1741: URL: https://github.com/apache/iceberg-python/pull/1741 This is a follow up item from a recent discussion on the mailing list[1], where the community decided that 401 response should be preferred over 419 response on token expiry. [1] mailing

Re: [PR] pass proxy configuration from environment vars to http client [iceberg]

2025-02-27 Thread via GitHub
akhilputhiry commented on PR #12406: URL: https://github.com/apache/iceberg/pull/12406#issuecomment-2689600902 > Can you confirm that adding ProxySupport on top of the existing code will work, e.g. if withProxy is used? > Yes it works, I had tested with `org.apache.iceberg.rest.HTTPCl

Re: [I] Docs: Add clear indicators for required fields in Spark syntax on CREATE TABLE. [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] commented on issue #9545: URL: https://github.com/apache/iceberg/issues/9545#issuecomment-2689398660 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Data: Handle case where partition location is missing for `TableMigrationUtil` [iceberg]

2025-02-27 Thread via GitHub
jshmchenxi commented on PR #12212: URL: https://github.com/apache/iceberg/pull/12212#issuecomment-2689591419 Hi @manuzhang @RussellSpitzer @ebyhr, just checking in on this PR. All feedback from the previous rounds has been addressed, and I believe it’s ready for the final review. I know you

Re: [PR] feat: make output file name of write task consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu commented on PR #1720: URL: https://github.com/apache/iceberg-python/pull/1720#issuecomment-2689583907 > Hey @sharkdtu Thanks for raising this. What would be the benefit of adding this counter to the output? I think it is unique without the counter. Sorry for not providing de

Re: [PR] feat: Add Issue Template [iceberg-rust]

2025-02-27 Thread via GitHub
jonathanc-n commented on PR #1008: URL: https://github.com/apache/iceberg-rust/pull/1008#issuecomment-2689582859 @liurenjie1024 added the appropriate fixes, thank you for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] spec: fix the wrong example in Geo Types [iceberg]

2025-02-27 Thread via GitHub
jiayuasu commented on PR #12421: URL: https://github.com/apache/iceberg/pull/12421#issuecomment-2689566002 @szehon-ho can you review it? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] spec: fix the wrong example in Geo Types [iceberg]

2025-02-27 Thread via GitHub
jiayuasu opened a new pull request, #12421: URL: https://github.com/apache/iceberg/pull/12421 ## Background According to the Iceberg Geo types spec, the CRS format follows `type:identifier` and the only allowed types are `srid` or `projjson`, no `ogc`. Whenever the spec says `O

Re: [PR] OpenAPI: Use more clear language in recommending error responses [iceberg]

2025-02-27 Thread via GitHub
sungwy commented on code in PR #12376: URL: https://github.com/apache/iceberg/pull/12376#discussion_r1974643193 ## open-api/rest-catalog-open-api.yaml: ## @@ -4566,7 +4568,9 @@ components: AuthenticationTimeoutResponse: description: -Credentials have timed

Re: [PR] Wrap variant in PrimitiveHoder so serialization can result same instance [iceberg]

2025-02-27 Thread via GitHub
flyrain commented on code in PR #12317: URL: https://github.com/apache/iceberg/pull/12317#discussion_r1974324857 ## api/src/main/java/org/apache/iceberg/types/PrimitiveHolder.java: ## @@ -21,7 +21,7 @@ import java.io.ObjectStreamException; import java.io.Serializable; -/** R

Re: [I] Docs: Add clear indicators for required fields in Spark syntax on CREATE TABLE. [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] closed issue #9545: Docs: Add clear indicators for required fields in Spark syntax on CREATE TABLE. URL: https://github.com/apache/iceberg/issues/9545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] tbl.append(df): schema validation of tbl & df during compares the order & data types [iceberg-python]

2025-02-27 Thread via GitHub
github-actions[bot] commented on issue #1088: URL: https://github.com/apache/iceberg-python/issues/1088#issuecomment-2689401600 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] Core: Refactor Table Metadata Tests [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] closed pull request #11947: Core: Refactor Table Metadata Tests URL: https://github.com/apache/iceberg/pull/11947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Core: Refactor Table Metadata Tests [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] commented on PR #11947: URL: https://github.com/apache/iceberg/pull/11947#issuecomment-2689398862 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Support bucket transform on multiple data columns [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] closed issue #5626: Support bucket transform on multiple data columns URL: https://github.com/apache/iceberg/issues/5626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Kafka Connect: Add delta writer support [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] commented on issue #10842: URL: https://github.com/apache/iceberg/issues/10842#issuecomment-2689398705 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2025-02-27 Thread via GitHub
github-actions[bot] commented on PR #11369: URL: https://github.com/apache/iceberg/pull/11369#issuecomment-2689398791 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Docs: Site updates for 1.8.1 [iceberg]

2025-02-27 Thread via GitHub
manuzhang commented on code in PR #12410: URL: https://github.com/apache/iceberg/pull/12410#discussion_r1974496241 ## site/docs/releases.md: ## @@ -72,6 +72,25 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.8.1 release + +Apache

Re: [PR] Docs: Site updates for 1.8.1 [iceberg]

2025-02-27 Thread via GitHub
manuzhang commented on code in PR #12410: URL: https://github.com/apache/iceberg/pull/12410#discussion_r1974496241 ## site/docs/releases.md: ## @@ -72,6 +72,25 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.8.1 release + +Apache

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-27 Thread via GitHub
aihuaxu commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1974489717 ## core/src/main/java/org/apache/iceberg/avro/AvroCustomOrderSchemaVisitor.java: ## @@ -27,13 +27,20 @@ import org.apache.iceberg.relocated.com.google.common.collect

Re: [PR] OpenAPI: Use more clear language in recommending error responses [iceberg]

2025-02-27 Thread via GitHub
flyrain commented on code in PR #12376: URL: https://github.com/apache/iceberg/pull/12376#discussion_r1974452617 ## open-api/rest-catalog-open-api.yaml: ## @@ -4566,7 +4568,9 @@ components: AuthenticationTimeoutResponse: description: -Credentials have timed

Re: [PR] Bump pyspark from 3.5.3 to 3.5.4 [iceberg-python]

2025-02-27 Thread via GitHub
dependabot[bot] closed pull request #1461: Bump pyspark from 3.5.3 to 3.5.4 URL: https://github.com/apache/iceberg-python/pull/1461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] Build: Bump pyspark from 3.5.3 to 3.5.5 [iceberg-python]

2025-02-27 Thread via GitHub
dependabot[bot] opened a new pull request, #1740: URL: https://github.com/apache/iceberg-python/pull/1740 Bumps [pyspark](https://github.com/apache/spark) from 3.5.3 to 3.5.5. Commits https://github.com/apache/spark/commit/7c29c664cdc9321205a98a14858aaf8daaa19db2";>7c29c66 Prep

Re: [PR] Bump pyspark from 3.5.3 to 3.5.4 [iceberg-python]

2025-02-27 Thread via GitHub
dependabot[bot] commented on PR #1461: URL: https://github.com/apache/iceberg-python/pull/1461#issuecomment-2689265240 Superseded by #1740. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] Build: Bump getdaft from 0.4.4 to 0.4.5 [iceberg-python]

2025-02-27 Thread via GitHub
dependabot[bot] opened a new pull request, #1739: URL: https://github.com/apache/iceberg-python/pull/1739 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.4.4 to 0.4.5. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [I] Reconsider handling of spaces in PartitionSpec$partitionToPath [iceberg]

2025-02-27 Thread via GitHub
bharos commented on issue #1479: URL: https://github.com/apache/iceberg/issues/1479#issuecomment-2689200126 Revisiting this issue, I notice even in V1.8 with Spark3.5 iceberg partition values are having "+" in the path instead of space. It can pose some problems, For example, let's say I

Re: [I] Alter table to generic types [iceberg]

2025-02-27 Thread via GitHub
jtavernier commented on issue #2791: URL: https://github.com/apache/iceberg/issues/2791#issuecomment-2689091883 We're facing the same challenges; this feature would be highly beneficial. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
lliangyu-lin commented on code in PR #12401: URL: https://github.com/apache/iceberg/pull/12401#discussion_r1974114780 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -494,22 +499,22 @@ public ViewMetadata build() { @VisibleForTesting static Lis

Re: [PR] Glue: Allow for assuming role for Glue [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on PR #1299: URL: https://github.com/apache/iceberg-python/pull/1299#issuecomment-2689007720 Since we undeprecated botocore session, we can close this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Include DELETE entries when inspecting [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on PR #1731: URL: https://github.com/apache/iceberg-python/pull/1731#issuecomment-2689012943 > I think we should make the same change to the _files, not so sure about partitions That's funny, was thinking the other way around. For `_files` it lists all the files of th

Re: [PR] Glue: Allow for assuming role for Glue [iceberg-python]

2025-02-27 Thread via GitHub
Fokko closed pull request #1299: Glue: Allow for assuming role for Glue URL: https://github.com/apache/iceberg-python/pull/1299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: make output file name of write task consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on PR #1720: URL: https://github.com/apache/iceberg-python/pull/1720#issuecomment-2688920222 Hey @sharkdtu Thanks for raising this. What would be the benefit of adding this counter to the output? I think it is unique without the counter. -- This is an automated message fro

Re: [PR] fix adlss properties processing [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on PR #1732: URL: https://github.com/apache/iceberg-python/pull/1732#issuecomment-2688907519 This is the equivalent code on the Java side of things: https://github.com/apache/iceberg/blob/a50ec923f3d928f67e2a4a361c0d1162341aa084/azure/src/main/java/org/apache/iceberg/azure/Az

Re: [PR] pass proxy configuration from environment vars to http client [iceberg]

2025-02-27 Thread via GitHub
adutra commented on PR #12406: URL: https://github.com/apache/iceberg/pull/12406#issuecomment-264965 Hi @akhilputhiry while I understand the problem I think there are a few concerns with this PR: First off, there is already some proxy support in `HTTPClient`: 1. `org.apache

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
lliangyu-lin commented on code in PR #12401: URL: https://github.com/apache/iceberg/pull/12401#discussion_r1974114780 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -494,22 +499,22 @@ public ViewMetadata build() { @VisibleForTesting static Lis

Re: [PR] fix adlss properties processing [iceberg-python]

2025-02-27 Thread via GitHub
ms32035 commented on PR #1732: URL: https://github.com/apache/iceberg-python/pull/1732#issuecomment-2688762349 That probably would be a question to @Fokko who authored #1520 I'm not even sure what the correct property specification is (if any), and what other rest catalogs may return

Re: [PR] AWS: Integrate S3 analytics accelerator library [iceberg]

2025-02-27 Thread via GitHub
jackye1995 commented on code in PR #12299: URL: https://github.com/apache/iceberg/pull/12299#discussion_r1974106727 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -640,12 +663,21 @@ public S3FileIOProperties(Map properties) { properti

Re: [PR] AWS: Integrate S3 analytics accelerator library [iceberg]

2025-02-27 Thread via GitHub
jackye1995 commented on code in PR #12299: URL: https://github.com/apache/iceberg/pull/12299#discussion_r1974106727 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -640,12 +663,21 @@ public S3FileIOProperties(Map properties) { properti

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-27 Thread via GitHub
danielcweeks commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1974017826 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java: ## @@ -200,7 +207,8 @@ static class PositionDeletesWriterFact

Re: [PR] AWS: Integrate S3 analytics accelerator library [iceberg]

2025-02-27 Thread via GitHub
jackye1995 commented on code in PR #12299: URL: https://github.com/apache/iceberg/pull/12299#discussion_r1974098939 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3URI.java: ## @@ -31,7 +31,7 @@ * * Note: Path-style access is deprecated and not supported by this implementa

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
amitgilad3 commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1974076443 ## tests/integration/test_add_files.py: ## @@ -229,6 +229,35 @@ def test_add_files_to_unpartitioned_table_raises_has_field_ids( tbl.add_files(file_p

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
vtk9 commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1974029646 ## pyiceberg/io/pyarrow.py: ## @@ -2464,38 +2464,37 @@ def _check_pyarrow_schema_compatible( _check_schema_compatible(requested_schema, provided_schema) -d

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-27 Thread via GitHub
danielcweeks commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1974017826 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java: ## @@ -200,7 +207,8 @@ static class PositionDeletesWriterFact

Re: [PR] REST auth error [iceberg]

2025-02-27 Thread via GitHub
adutra commented on code in PR #12415: URL: https://github.com/apache/iceberg/pull/12415#discussion_r1973976074 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -258,7 +258,9 @@ public void initialize(String name, Map unresolved) { config =

[PR] Core: Code cleanup around TestTable and TestTableOperations [iceberg]

2025-02-27 Thread via GitHub
gaborkaszab opened a new pull request, #12419: URL: https://github.com/apache/iceberg/pull/12419 1: TestTableOperations already contains a member for the table name. It's redundant to have the same as a separate constuctor parameter for TestTable. 2: The 2 constructors for TestTableOp

Re: [PR] REST auth error [iceberg]

2025-02-27 Thread via GitHub
adutra commented on code in PR #12415: URL: https://github.com/apache/iceberg/pull/12415#discussion_r1974012751 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -258,7 +258,9 @@ public void initialize(String name, Map unresolved) { config =

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
vtk9 commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1974027542 ## pyiceberg/io/pyarrow.py: ## @@ -2466,36 +2466,41 @@ def _check_pyarrow_schema_compatible( def parquet_files_to_data_files(io: FileIO, table_metadata: TableMet

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
vtk9 commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1974026972 ## tests/integration/test_add_files.py: ## @@ -229,6 +229,35 @@ def test_add_files_to_unpartitioned_table_raises_has_field_ids( tbl.add_files(file_paths=f

[PR] Build: Ignore docker folder in CI [iceberg]

2025-02-27 Thread via GitHub
manuzhang opened a new pull request, #12417: URL: https://github.com/apache/iceberg/pull/12417 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-02-27 Thread via GitHub
andyglow opened a new pull request, #12418: URL: https://github.com/apache/iceberg/pull/12418 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] pass proxy configuration from environment vars to http client [iceberg]

2025-02-27 Thread via GitHub
akhilputhiry commented on PR #12406: URL: https://github.com/apache/iceberg/pull/12406#issuecomment-2688573051 @adutra Could you please take a look at this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] I do not understand the partition error: ValueError: Could not find in old schema: 2: {field}: identity(2) [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu commented on issue #1100: URL: https://github.com/apache/iceberg-python/issues/1100#issuecomment-2688544747 i see something similar in https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/api.md#partition-evolution -- This is an automated message from the Apache Git

Re: [PR] REST auth error [iceberg]

2025-02-27 Thread via GitHub
adutra commented on code in PR #12415: URL: https://github.com/apache/iceberg/pull/12415#discussion_r1973976074 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -258,7 +258,9 @@ public void initialize(String name, Map unresolved) { config =

Re: [I] DOCS: Clearly document unsupported Iceberg Features [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu commented on issue #1109: URL: https://github.com/apache/iceberg-python/issues/1109#issuecomment-2688535332 This is partial done by https://iceberg.apache.org/status/ which we should update after 0.9.0 is officially released -- This is an automated message from the Apache Git

Re: [I] Merge into / Upsert [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu commented on issue #402: URL: https://github.com/apache/iceberg-python/issues/402#issuecomment-2688531149 yes we have plans to improve upsert functionalities. The current implementation was a first pass to support the minimal functionalities of upsert :) I can open a new

Re: [I] Getting "offset overflow while concatenating arrays" Error when writing to iceberg [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu commented on issue #1733: URL: https://github.com/apache/iceberg-python/issues/1733#issuecomment-2688526785 What version of the library are you using? there was a similar issue with the same error message #1491 which was fixed with #1555 can you use the latest version to c

Re: [PR] Remove unused argument [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu merged PR #1734: URL: https://github.com/apache/iceberg-python/pull/1734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Remove unused argument [iceberg-python]

2025-02-27 Thread via GitHub
kevinjqliu commented on code in PR #1734: URL: https://github.com/apache/iceberg-python/pull/1734#discussion_r1973946050 ## pyiceberg/table/__init__.py: ## @@ -1716,7 +1715,6 @@ def plan_files(self) -> Iterable[FileScanTask]: self.io,

[PR] Ignore apt-get update error when building `iceberg-rest-fixture` image [iceberg]

2025-02-27 Thread via GitHub
manuzhang opened a new pull request, #12416: URL: https://github.com/apache/iceberg/pull/12416 `apt-get update` could fail to fetch updates from some archives and break the build. ``` Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c set -xeu && groupadd

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1973903275 ## pyiceberg/io/pyarrow.py: ## @@ -2466,36 +2466,41 @@ def _check_pyarrow_schema_compatible( def parquet_files_to_data_files(io: FileIO, table_metadata: TableMe

Re: [PR] Use delimited column names in `CreateChangelogViewProcedure` [iceberg]

2025-02-27 Thread via GitHub
andyglow closed pull request #12322: Use delimited column names in `CreateChangelogViewProcedure` URL: https://github.com/apache/iceberg/pull/12322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] parallelize `add_files` [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on code in PR #1717: URL: https://github.com/apache/iceberg-python/pull/1717#discussion_r1973893895 ## pyiceberg/io/pyarrow.py: ## @@ -2464,38 +2464,37 @@ def _check_pyarrow_schema_compatible( _check_schema_compatible(requested_schema, provided_schema) -

Re: [PR] fix adlss properties processing [iceberg-python]

2025-02-27 Thread via GitHub
Fokko commented on PR #1732: URL: https://github.com/apache/iceberg-python/pull/1732#issuecomment-2688404050 @ms32035 Thank you for raising this. Can you also run `make lint`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Parquet: Implement Variant writers [iceberg]

2025-02-27 Thread via GitHub
rdblue commented on code in PR #12323: URL: https://github.com/apache/iceberg/pull/12323#discussion_r1973887545 ## core/src/main/java/org/apache/iceberg/variants/VariantVisitor.java: ## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[PR] REST auth error [iceberg]

2025-02-27 Thread via GitHub
aihuaxu opened a new pull request, #12415: URL: https://github.com/apache/iceberg/pull/12415 REST auth error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Build: Bump mkdocstrings-python from 1.16.1 to 1.16.2 [iceberg-python]

2025-02-27 Thread via GitHub
Fokko merged PR #1729: URL: https://github.com/apache/iceberg-python/pull/1729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] OpenAPI: Use more clear language in recommending error responses [iceberg]

2025-02-27 Thread via GitHub
dimas-b commented on code in PR #12376: URL: https://github.com/apache/iceberg/pull/12376#discussion_r1973840064 ## open-api/rest-catalog-open-api.yaml: ## @@ -4566,7 +4568,9 @@ components: AuthenticationTimeoutResponse: description: -Credentials have timed

Re: [PR] Core: Apply correct metric configs in GenericAppenderFactory [iceberg]

2025-02-27 Thread via GitHub
XBaith commented on PR #12366: URL: https://github.com/apache/iceberg/pull/12366#issuecomment-2688338232 Thank you so much for the review, @pvary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
nastra commented on code in PR #12401: URL: https://github.com/apache/iceberg/pull/12401#discussion_r1973802271 ## core/src/test/java/org/apache/iceberg/view/TestViewMetadata.java: ## @@ -396,6 +397,28 @@ public void viewVersionHistoryIsCorrectlyRetained() { .hasMessage

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
nastra commented on code in PR #12401: URL: https://github.com/apache/iceberg/pull/12401#discussion_r1973801663 ## core/src/test/java/org/apache/iceberg/view/TestViewMetadata.java: ## @@ -396,6 +397,28 @@ public void viewVersionHistoryIsCorrectlyRetained() { .hasMessage

Re: [PR] Core: Ensure current and newly added view versions are retained in ViewMetadata build [iceberg]

2025-02-27 Thread via GitHub
nastra commented on code in PR #12401: URL: https://github.com/apache/iceberg/pull/12401#discussion_r1973800528 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -494,22 +499,22 @@ public ViewMetadata build() { @VisibleForTesting static List expi

Re: [PR] Core: Interface changes for separating rewrite planner and runner [iceberg]

2025-02-27 Thread via GitHub
pvary commented on PR #12306: URL: https://github.com/apache/iceberg/pull/12306#issuecomment-2688197241 Thanks @stevenzwu and @RussellSpitzer! I will be OOO next week, so the next PR will come after that -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [I] [Spark] Cannot append to Glue table - StorageDescriptor#InputFormat cannot be null for table [iceberg]

2025-02-27 Thread via GitHub
yruslan commented on issue #5565: URL: https://github.com/apache/iceberg/issues/5565#issuecomment-2688172078 I second the question. The issue doesn't look like solved, and it requires a workaround. Here is my workaround based on @t0ma-sz 's one: ```scala def doesTableExist(ful

[I] [Feature Request] Add Writer Support for Table-Compatible Parquet Files [iceberg-python]

2025-02-27 Thread via GitHub
andormarkus-alcd opened a new issue, #1737: URL: https://github.com/apache/iceberg-python/issues/1737 ### Feature Request / Improvement ## Problem Statement **I'm happy to submit a PR to implement this feature.** PyIceberg currently provides functionality to add existing

Re: [I] Appending data to a partitioned table by pyiceberg [iceberg]

2025-02-27 Thread via GitHub
raphaelauv commented on issue #12399: URL: https://github.com/apache/iceberg/issues/12399#issuecomment-2688156278 it's part of pyiceberg 0.9 -> https://github.com/apache/iceberg-python/issues/208 @yaelperl1234 could you please close this issue , thanks -- This is an automated mess

Re: [PR] Auth Manager API part 6: API enablement [iceberg]

2025-02-27 Thread via GitHub
nastra commented on code in PR #12197: URL: https://github.com/apache/iceberg/pull/12197#discussion_r1965346332 ## aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3RestSigner.java: ## @@ -86,8 +86,10 @@ public class TestS3RestSigner { MinioUtil.createContainer(CRE

Re: [PR] feat: make output data path of table with identity timestamp partition consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sungwy commented on PR #1736: URL: https://github.com/apache/iceberg-python/pull/1736#issuecomment-2687971962 Hi @sharkdtu thank you for working on this PR! 😊 I think consistency is great, but Iceberg currently does not require that we guarantee consistent paths (unlike Hive style par

Re: [PR] Spark 3.5: Fix RewriteDataFiles with partial progress enabled and max-failed-commits larger than total-file-group [iceberg]

2025-02-27 Thread via GitHub
manuzhang commented on PR #12120: URL: https://github.com/apache/iceberg/pull/12120#issuecomment-2687964763 @nastra any more comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] The identity partition path of timestamp type is inconsistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sungwy commented on issue #1735: URL: https://github.com/apache/iceberg-python/issues/1735#issuecomment-2687962641 Hi @sharkdtu my understanding is that Iceberg does not make any guarantees on the paths of the data files, as it relies on links to connect data files of a snapshot together (

[I] Partition info is not displayed in job description for RewritePositionDeleteFilesSparkAction [iceberg]

2025-02-27 Thread via GitHub
manuzhang opened a new issue, #12414: URL: https://github.com/apache/iceberg/issues/12414 ### Feature Request / Improvement ![Image](https://github.com/user-attachments/assets/d0087db5-9d58-4380-b98b-82cda1414229) It's due to `PartitionData` being [coerced into `StructProjecti

Re: [PR] feat: make output data path of table with identity timestamp partition consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu commented on PR #1736: URL: https://github.com/apache/iceberg-python/pull/1736#issuecomment-2687860095 @Fokko Could you please take a look at this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[PR] feat: make output data path of table with identity timestamp partition consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu opened a new pull request, #1736: URL: https://github.com/apache/iceberg-python/pull/1736 Resolves: #1735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] feat: make output file name of write task consistent with java api [iceberg-python]

2025-02-27 Thread via GitHub
sharkdtu commented on PR #1720: URL: https://github.com/apache/iceberg-python/pull/1720#issuecomment-2687854590 @Fokko Could you please take a look at this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

  1   2   >