Re: [I] Can we make commits inside compaction jobs with partial-progress.enabled sequential to avoid CommitFailedException? [iceberg]

ajantha-bhat commented on issue #9687: URL: https://github.com/apache/iceberg/issues/9687#issuecomment-1935477695 I think in iceberg-Kafaka-connect we have a logic to sequentially commit the multiple commits using commit coordinator (cc: @bryanck). Maybe we can reuse the code or extract it

Re: [I] Conversion between date/timestamp(tz) and integer for avro reader/writer [iceberg-python]

Fokko commented on issue #398: URL: https://github.com/apache/iceberg-python/issues/398#issuecomment-1935466554 Thanks @jqin61 🙌 If we can split out this kind of task into a separate PR would be preferred. This way we keep the partitioned writes PR a bit manageable :) -- This is an auto

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1935456427 Oh by the way, I will update the PR to use single quote on the statement as double quote has a special meaning on some databases (like PostgreSQL). I'm on it. -- This is an automated

Re: [PR] Downgrade to the previous artifact upload action [iceberg-python]

Fokko commented on PR #396: URL: https://github.com/apache/iceberg-python/pull/396#issuecomment-1935450194 @HonahX Thanks for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Downgrade to the previous artifact upload action [iceberg-python]

Fokko merged PR #396: URL: https://github.com/apache/iceberg-python/pull/396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump getdaft from 0.2.12 to 0.2.13 [iceberg-python]

Fokko merged PR #399: URL: https://github.com/apache/iceberg-python/pull/399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump griffe from 0.40.0 to 0.40.1 [iceberg-python]

Fokko merged PR #400: URL: https://github.com/apache/iceberg-python/pull/400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1935445632 @ajantha-bhat @nastra @rdblue @danielcweeks I updated the PR with: - support old SQL schema (not updated) - auto update (by configuration, disabled by default) - new tests for

Re: [I] Iceberg table not able to read data from S3 after few hours using Athena . [iceberg]

ajsalunkhe commented on issue #9684: URL: https://github.com/apache/iceberg/issues/9684#issuecomment-1935443901 We are creating iceberg table using below sql query using Athena in Glue Catalog: **CREATE TABLE db_name.table_name(col1 string, col2 string, col3 string) PARTITIONED BY

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

ggershinsky commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1483928229 ## api/src/main/java/org/apache/iceberg/encryption/EncryptingFileIO.java: ## @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

ggershinsky commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1483931060 ## core/src/main/java/org/apache/iceberg/ManifestFiles.java: ## @@ -345,34 +345,24 @@ private static ManifestFile copyManifestInternal( return writer.toManifes

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

ggershinsky commented on PR #9592: URL: https://github.com/apache/iceberg/pull/9592#issuecomment-1935414740 @rdblue Thanks for the PR. I am about to complete applying it to the e2e manifest/list/table encryption. So far, everything works well. I'll send updated patches next week. -- This

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

ajantha-bhat commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483911297 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483884220 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributo

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483877299 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -68,21 +61,27 @@ default Integer currentSchemaId() { return currentSchemaId; } + @Value

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

ajantha-bhat commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483877815 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

ajantha-bhat commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483877581 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483877299 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -68,21 +61,27 @@ default Integer currentSchemaId() { return currentSchemaId; } + @Value

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483876244 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributo

Re: [PR] Try Out Creating a PR Into the docs Branch [iceberg]

nathanwilk7 commented on PR #9692: URL: https://github.com/apache/iceberg/pull/9692#issuecomment-1935292703 Thanks, I'll close this PR as well so it's not showing up anywhere -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Try Out Creating a PR Into the docs Branch [iceberg]

nathanwilk7 closed pull request #9692: Try Out Creating a PR Into the docs Branch URL: https://github.com/apache/iceberg/pull/9692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[I] Getting Original Schema of a DataFile in a FileScanTask? [iceberg-python]

srilman opened a new issue, #401: URL: https://github.com/apache/iceberg-python/issues/401 ### Question Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during `FileTableScan.plan_files`? This is useful to determi

Re: [PR] Try Out Creating a PR Into the docs Branch [iceberg]

bitsondatadev commented on PR #9692: URL: https://github.com/apache/iceberg/pull/9692#issuecomment-1935288333 Nailed it @nathanwilk7! btw, you can also mark this as a "Draft" PR in GitHub to indicate it's not ready yet. -- This is an automated message from the Apache Git Service. To res

Re: [PR] Support for pushdown like filter (endsWith and contains) [iceberg]

yabola commented on PR #9683: URL: https://github.com/apache/iceberg/pull/9683#issuecomment-1935287304 > @yabola This is very cool, could we break up the PR though for easier review? Let me take a deeper look at the code before I propose a way to break it up into separate PRs. On the surfac

[PR] Try Out Creating a PR Into the docs Branch [iceberg]

nathanwilk7 opened a new pull request, #9692: URL: https://github.com/apache/iceberg/pull/9692 @bitsondatadev this is just a test branch to see if my git workflow matches what you expect it too. I'm not familiar with the orphaned `docs` branch setup you're using, so let me know if there's s

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1483788053 ## open-api/rest-catalog-open-api.yaml: ## @@ -212,6 +212,34 @@ paths: schema: type: string example: "accounting%1Ftax" +- na

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1483788053 ## open-api/rest-catalog-open-api.yaml: ## @@ -212,6 +212,34 @@ paths: schema: type: string example: "accounting%1Ftax" +- na

[PR] Pass input file into iterators to get the file name [iceberg]

javsanbel2 opened a new pull request, #9691: URL: https://github.com/apache/iceberg/pull/9691 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [I] [SPARK-3] Extend iceberg Catalog interface with namespace operations [iceberg]

github-actions[bot] commented on issue #1039: URL: https://github.com/apache/iceberg/issues/1039#issuecomment-1935126803 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] official site none offer get start and in action document [iceberg]

github-actions[bot] commented on issue #1009: URL: https://github.com/apache/iceberg/issues/1009#issuecomment-1935126720 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] lower_bounds and upper_bounds are not collected for columns in map type [iceberg]

github-actions[bot] commented on issue #978: URL: https://github.com/apache/iceberg/issues/978#issuecomment-1935126632 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Implement the split logic in DataTask [iceberg]

github-actions[bot] commented on issue #976: URL: https://github.com/apache/iceberg/issues/976#issuecomment-1935126612 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] While alter table drop partition column, the error message is not friendly. [iceberg]

github-actions[bot] closed issue #714: While alter table drop partition column, the error message is not friendly. URL: https://github.com/apache/iceberg/issues/714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Support for Apache Beam I/O [iceberg]

github-actions[bot] commented on issue #693: URL: https://github.com/apache/iceberg/issues/693#issuecomment-1935126367 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Partition Spec Performance [iceberg]

github-actions[bot] closed issue #692: Partition Spec Performance URL: https://github.com/apache/iceberg/issues/692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] Add an action to rewrite equality deletes as position deletes [iceberg]

github-actions[bot] commented on issue #1026: URL: https://github.com/apache/iceberg/issues/1026#issuecomment-1935126776 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Add a merge-based row filter for equality deletes [iceberg]

github-actions[bot] commented on issue #1024: URL: https://github.com/apache/iceberg/issues/1024#issuecomment-1935126749 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [Update notification follow up] We should publish update events only when a transaction succeeds [iceberg]

github-actions[bot] commented on issue #998: URL: https://github.com/apache/iceberg/issues/998#issuecomment-1935126700 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] [Update notification follow up] Add remaining events for SnapshotManager [iceberg]

github-actions[bot] commented on issue #997: URL: https://github.com/apache/iceberg/issues/997#issuecomment-1935126682 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] New warnings when building Iceberg [iceberg]

github-actions[bot] commented on issue #988: URL: https://github.com/apache/iceberg/issues/988#issuecomment-1935126659 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Write stream of unordered rows into partitioned table causes "Already closed files for partition" [iceberg]

github-actions[bot] commented on issue #717: URL: https://github.com/apache/iceberg/issues/717#issuecomment-1935126449 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Write stream of unordered rows into partitioned table causes "Already closed files for partition" [iceberg]

github-actions[bot] closed issue #717: Write stream of unordered rows into partitioned table causes "Already closed files for partition" URL: https://github.com/apache/iceberg/issues/717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] [Spark-3] Add missing abortStagedChanges support for StagedSparkTable [iceberg]

github-actions[bot] commented on issue #698: URL: https://github.com/apache/iceberg/issues/698#issuecomment-1935126394 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] While alter table drop partition column, the error message is not friendly. [iceberg]

github-actions[bot] commented on issue #714: URL: https://github.com/apache/iceberg/issues/714#issuecomment-1935126422 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] [Spark-3] Add missing abortStagedChanges support for StagedSparkTable [iceberg]

github-actions[bot] closed issue #698: [Spark-3] Add missing abortStagedChanges support for StagedSparkTable URL: https://github.com/apache/iceberg/issues/698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Support for Apache Beam I/O [iceberg]

github-actions[bot] closed issue #693: Support for Apache Beam I/O URL: https://github.com/apache/iceberg/issues/693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [I] Partition Spec Performance [iceberg]

github-actions[bot] commented on issue #692: URL: https://github.com/apache/iceberg/issues/692#issuecomment-1935126338 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

[PR] Build: Bump griffe from 0.40.0 to 0.40.1 [iceberg-python]

dependabot[bot] opened a new pull request, #400: URL: https://github.com/apache/iceberg-python/pull/400 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 0.40.0 to 0.40.1. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

[PR] Build: Bump getdaft from 0.2.12 to 0.2.13 [iceberg-python]

dependabot[bot] opened a new pull request, #399: URL: https://github.com/apache/iceberg-python/pull/399 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.12 to 0.2.13. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

[PR] OpenAPI: Spec updates for statistics [iceberg]

mrcnc opened a new pull request, #9690: URL: https://github.com/apache/iceberg/pull/9690 As a follow up for https://github.com/apache/iceberg/pull/9564 I added the statistics files to the table metadata. Then I noticed the partition statistics files were also missing, so I added them

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

szehon-ho commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1483639368 ## format/spec.md: ## @@ -1117,7 +1117,17 @@ Partition specs are serialized as a JSON object with the following fields: |**`spec-id`**|`JSON int`|`0`| |**`fields`*

Re: [I] Updating a property map in a iceberg table [iceberg]

rdblue commented on issue #9659: URL: https://github.com/apache/iceberg/issues/9659#issuecomment-1934991077 I don't think this is a good idea. We want to avoid keeping complex state in table properties and use cases involving complex state in properties are often not a good use of propertie

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

szehon-ho commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1483630271 ## format/spec.md: ## @@ -1130,14 +1140,11 @@ Each partition field in the fields list is stored as an object. See the table fo |**`hour`**|`JSON string: "hour"`|`"h

Re: [I] Conversion between date/timestamp(tz) and integer for avro reader/writer [iceberg-python]

jqin61 commented on issue #398: URL: https://github.com/apache/iceberg-python/issues/398#issuecomment-1934984862 Thank you for the explanation! Will move forward with the first option as instructed. -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Conversion between date/timestamp(tz) and integer for avro reader/writer [iceberg-python]

jqin61 closed issue #398: Conversion between date/timestamp(tz) and integer for avro reader/writer URL: https://github.com/apache/iceberg-python/issues/398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1483627670 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -205,38 +208,74 @@ public String toString() { } public static class TimestampType extends Primiti

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1483627324 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -205,38 +208,74 @@ public String toString() { } public static class TimestampType extends Primiti

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1483626438 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -46,8 +47,10 @@ private Types() {} .put(DoubleType.get().toString(), DoubleType.get())

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1483624801 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -564,14 +569,26 @@ private static String sanitizeDate(int days, int today) { return "

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1483623055 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -501,18 +531,22 @@ public Literal to(Type type) { return (Literal) new TimeLiteral(t

Re: [PR] Core: Add property to prevent loss of view representation when replacing a view [iceberg]

rdblue commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1483619256 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -450,6 +452,15 @@ public ViewMetadata build() { ViewProperties.VERSION_HISTORY_SIZE,

Re: [PR] Core: Add property to prevent loss of view representation when replacing a view [iceberg]

rdblue commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1483615535 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -510,5 +521,29 @@ static List updateHistory(List history, Set< private Stream changes(Class

Re: [I] Conversion between date/timestamp(tz) and integer for avro reader/writer [iceberg-python]

Fokko commented on issue #398: URL: https://github.com/apache/iceberg-python/issues/398#issuecomment-1934910609 Thanks for raising this @jqin61. The date is a logical type, where the integer represents the days since 1-1-1970. To public APIs, we should accept `datetime.date` for conve

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1483562486 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -149,4 +167,20 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1483559483 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -149,4 +167,20 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1483557926 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -115,7 +124,16 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1483557926 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -115,7 +124,16 @@ case class RewriteViewCommands(sp

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1483553753 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,14 +18,116 @@ */ package org.apache.iceberg.jdbc; +import static org.assertj.core.api.A

[I] java.lang.IllegalArgumentException: requirement failed: length (-6235972) cannot be smaller than -1 [iceberg]

rjayapalan opened a new issue, #9689: URL: https://github.com/apache/iceberg/issues/9689 ### Apache Iceberg version 1.4.2 ### Query engine Spark ### Please describe the bug 🐞 I am aware of this similar issue that was addressed as part of iceberg 1.4.1 relea

[I] conversion between date/timestamp(tz) and integer for avro reader/writer [iceberg-python]

jqin61 opened a new issue, #398: URL: https://github.com/apache/iceberg-python/issues/398 ### Question Hi I was trying to understand whether there is any special design initiative for: 1. The Python representation of partition of datetime-related type is integer. For example: `

Re: [I] Reorder query engines in docs as alphanumeric. [iceberg]

bitsondatadev commented on issue #9681: URL: https://github.com/apache/iceberg/issues/9681#issuecomment-1934810341 > I don't think vendors like AWS EMR or Cloudera should be listed here. They may provide engines, catalogs or compaction services that integrate with Iceberg. Perhaps th

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

szehon-ho commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483497724 ## core/src/main/java/org/apache/iceberg/BaseMetadata.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

szehon-ho commented on code in PR #9682: URL: https://github.com/apache/iceberg/pull/9682#discussion_r1483494509 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -68,21 +61,27 @@ default Integer currentSchemaId() { return currentSchemaId; } + @Va

Re: [I] TestDataFrameWrites fails due to OOM errors with ORC [iceberg]

aokolnychyi commented on issue #964: URL: https://github.com/apache/iceberg/issues/964#issuecomment-1934800820 No longer applies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] TestDataFrameWrites fails due to OOM errors with ORC [iceberg]

aokolnychyi closed issue #964: TestDataFrameWrites fails due to OOM errors with ORC URL: https://github.com/apache/iceberg/issues/964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] feat(catalog): add initial rest catalog impl [iceberg-go]

wolfeidau commented on code in PR #58: URL: https://github.com/apache/iceberg-go/pull/58#discussion_r1483479148 ## catalog/catalog.go: ## @@ -47,19 +52,136 @@ func WithAwsConfig(cfg aws.Config) Option { } } +func WithCredential(cred string) Option { Review Comment:

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.11 [iceberg]

dependabot[bot] commented on PR #9472: URL: https://github.com/apache/iceberg/pull/9472#issuecomment-1934709260 Superseded by #9688. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.11 [iceberg]

dependabot[bot] closed pull request #9472: Build: Bump slf4j from 1.7.36 to 2.0.11 URL: https://github.com/apache/iceberg/pull/9472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] Build: Bump slf4j from 1.7.36 to 2.0.12 [iceberg]

dependabot[bot] opened a new pull request, #9688: URL: https://github.com/apache/iceberg/pull/9688 Bumps `slf4j` from 1.7.36 to 2.0.12. Updates `org.slf4j:slf4j-api` from 1.7.36 to 2.0.12 Updates `org.slf4j:slf4j-simple` from 1.7.36 to 2.0.12 Dependabot will resolve any c

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.4 to 3.14.5 [iceberg]

Fokko merged PR #9570: URL: https://github.com/apache/iceberg/pull/9570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] Core: complete FileScanTaskParser for other FileScanTask implementation classes (like StaticDataTask) [iceberg]

stevenzwu commented on issue #9597: URL: https://github.com/apache/iceberg/issues/9597#issuecomment-1934568874 @nastra @aokolnychyi any feedback on the proposal of adding a `StructParser` JSON serializer? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] AWS: Add url-connection-client to aws-bundle [iceberg]

nastra commented on PR #9685: URL: https://github.com/apache/iceberg/pull/9685#issuecomment-1934536087 I'll hold off merging this because there might be a separate issue when bundling this dependency according to @danielcweeks -- This is an automated message from the Apache Git Service.

[I] Can we make commits inside compaction jobs with partial-progress.enabled sequential to avoid CommitFailedException? [iceberg]

paulpaul1076 opened a new issue, #9687: URL: https://github.com/apache/iceberg/issues/9687 ### Feature Request / Improvement From what I understand, if a compaction job compacts a lot of small files, and uses `partial-progress.enabled=true`, there are situations, when file groups get

Re: [I] RewriteManifest with more options [iceberg]

zachdisc commented on issue #9615: URL: https://github.com/apache/iceberg/issues/9615#issuecomment-1934514656 Looking at the base API interface and core implementation, there is a [ClusterBy](https://github.com/apache/iceberg/blob/90d1c90b6e6f26fdfe7c0c6c09a1ecb2fc2b3f2a/core/src/main/java/o

Re: [I] Reorder query engines in docs as alphanumeric. [iceberg]

scottteal commented on issue #9681: URL: https://github.com/apache/iceberg/issues/9681#issuecomment-1934476378 Would you take the same approach for catalogs with "Community Catalogs" (Hive Metastore, Nessie, etc.) and "Enterprise Catalogs" (Amazon DynamoDB, AWS Glue Data Catalog, Dremio Arc

Re: [PR] Support for pushdown like filter (endsWith and contains) [iceberg]

amogh-jahagirdar commented on PR #9683: URL: https://github.com/apache/iceberg/pull/9683#issuecomment-1934467372 Also, I'll need to think more if we can actually support this for delete files. If not, this will need to only be applied for CoW tables. For example, for agg pushdown, we don't

Re: [PR] Support for pushdown like filter (endsWith and contains) [iceberg]

amogh-jahagirdar commented on PR #9683: URL: https://github.com/apache/iceberg/pull/9683#issuecomment-1934428177 @yabola This is very cool, could we break up the PR though for easier review? Let me take a deeper look at the code before I propose a way to break it up into separate PRs. On th

Re: [PR] Fix: add required rust version in cargo.toml [iceberg-rust]

dp-0 commented on code in PR #193: URL: https://github.com/apache/iceberg-rust/pull/193#discussion_r1482309823 ## Cargo.toml: ## @@ -26,6 +26,7 @@ homepage = "https://rust.iceberg.apache.org/"; repository = "https://github.com/apache/iceberg-rust"; license = "Apache-2.0" +ru

Re: [PR] detect breaking changes [iceberg-python]

syun64 commented on PR #394: URL: https://github.com/apache/iceberg-python/pull/394#issuecomment-1934357519 An issue I'm currently investigating is that 58 breaking changes are being reported between 0.5.1 and 0e273ae when the test is run in my local environment. When test_api.py was

Re: [PR] Docs: Fix hidden-partition-animation not showing [iceberg]

munabedan commented on PR #9686: URL: https://github.com/apache/iceberg/pull/9686#issuecomment-1934356149 You are welcome @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Docs: Fix hidden-partition-animation not showing [iceberg]

Fokko merged PR #9686: URL: https://github.com/apache/iceberg/pull/9686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[PR] Docs: Fix hidden-partition-animation not showing [iceberg]

munabedan opened a new pull request, #9686: URL: https://github.com/apache/iceberg/pull/9686 This commit should fix hidden partition animation on https://iceberg.apache.org/ homepage Issue: https://github.com/apache/iceberg/issues/9643#issuecomment-1926369554 ![hiden-partitio

Re: [PR] Support merge manifests on writes [iceberg-python]

Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1482859941 ## pyiceberg/table/__init__.py: ## @@ -944,7 +949,8 @@ def append(self, df: pa.Table) -> None: if len(self.spec().fields) > 0: raise ValueError

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

paulpaul1076 commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-1933903982 Basically I had a streaming job that was streaming small files. Then I stopped it, tried compacting, and it failed with these content-length exceptions. I'll try to find some fr

Re: [PR] open-api: Use openapi-generator-gradle-plugin for validating specification [iceberg]

Fokko merged PR #9344: URL: https://github.com/apache/iceberg/pull/9344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[PR] Downgrade to the previous artifact upload action [iceberg-python]

Fokko opened a new pull request, #396: URL: https://github.com/apache/iceberg-python/pull/396 With the new version, you cannot merge the different runs anymore into a single zip: https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes Which is quite annoying si

Re: [I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

Fokko commented on issue #368: URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1933869341 I would love that, and this is what I suggested in https://github.com/apache/iceberg-python/issues/368#issuecomment-1928020308 -- This is an automated message from the Apache Gi

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.31.0 [iceberg]

dependabot[bot] commented on PR #9635: URL: https://github.com/apache/iceberg/pull/9635#issuecomment-1933769794 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.31.0 [iceberg]

nastra closed pull request #9635: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.31.0 URL: https://github.com/apache/iceberg/pull/9635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Iceberg table not able to read data from S3 after few hours using Athena . [iceberg]

nastra commented on issue #9684: URL: https://github.com/apache/iceberg/issues/9684#issuecomment-1933759635 @ajsalunkhe it's not clear from the description whether this is an Iceberg or an Athena-related issue. Can you elaborate and add a few more details which queries you ran exactly that

  1   2   >