Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826493625 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } + @Test

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11444: URL: https://github.com/apache/iceberg/pull/11444#discussion_r1826485532 ## api/src/test/java/org/apache/iceberg/TestSchema.java: ## @@ -64,27 +65,77 @@ public class TestSchema { .withWriteDefault("--") .build

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826477524 ## api/src/main/java/org/apache/iceberg/types/Type.java: ## @@ -92,6 +93,10 @@ default boolean isListType() { return false; } + default boolean isVariantTy

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826477007 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,8 @@ class Identity implements Transform { */ @Deprecated public static I

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
szehon-ho commented on PR #11437: URL: https://github.com/apache/iceberg/pull/11437#issuecomment-2452749792 Merged, thanks @dramaticlly ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Bad Table Properties cause commit failure [iceberg]

2024-11-01 Thread via GitHub
szehon-ho closed issue #11435: Bad Table Properties cause commit failure URL: https://github.com/apache/iceberg/issues/11435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
szehon-ho merged PR #11437: URL: https://github.com/apache/iceberg/pull/11437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Integration tests performance degradation [iceberg-python]

2024-11-01 Thread via GitHub
github-actions[bot] closed issue #604: Integration tests performance degradation URL: https://github.com/apache/iceberg-python/issues/604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Integration tests performance degradation [iceberg-python]

2024-11-01 Thread via GitHub
github-actions[bot] commented on issue #604: URL: https://github.com/apache/iceberg-python/issues/604#issuecomment-2452746637 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [PR] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10341: URL: https://github.com/apache/iceberg/pull/10341#issuecomment-2452745612 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] AWS: Change S3FileIO to use SHA1 based checksums [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10293: URL: https://github.com/apache/iceberg/pull/10293#issuecomment-2452745563 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10284: URL: https://github.com/apache/iceberg/pull/10284#issuecomment-2452745532 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark 3.5: Add validation to SparkConfParser [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10315: URL: https://github.com/apache/iceberg/pull/10315#issuecomment-2452745576 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Flink Table Maintenance [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on issue #10264: URL: https://github.com/apache/iceberg/issues/10264#issuecomment-2452745491 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Spark: Add SparkSQLProperty to control split-size [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10336: URL: https://github.com/apache/iceberg/pull/10336#issuecomment-2452745597 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10280: URL: https://github.com/apache/iceberg/pull/10280#issuecomment-2452745521 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Assign fresh IDs to view schema [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10253: URL: https://github.com/apache/iceberg/pull/10253#issuecomment-2452745478 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Iceberg.engine.hive.enabled Conf is not honouring for HIVE CATALOG #10286 [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10287: URL: https://github.com/apache/iceberg/pull/10287#issuecomment-2452745549 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Support rewriting manifests on branches [iceberg]

2024-11-01 Thread via GitHub
github-actions[bot] commented on PR #10265: URL: https://github.com/apache/iceberg/pull/10265#issuecomment-2452745503 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Flink: FLIP-27 IcebergSource builder missed a couple of configs compared to old FlinkSource: expose locality and plan parallelism [iceberg]

2024-11-01 Thread via GitHub
liuml07 commented on code in PR #10957: URL: https://github.com/apache/iceberg/pull/10957#discussion_r1826390148 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -482,6 +482,10 @@ public IcebergSource build() { } contex

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11444: URL: https://github.com/apache/iceberg/pull/11444#discussion_r1826348277 ## api/src/test/java/org/apache/iceberg/TestSchema.java: ## @@ -21,29 +21,30 @@ import static org.assertj.core.api.Assertions.assertThatCode; import static org.asse

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11444: URL: https://github.com/apache/iceberg/pull/11444#discussion_r1826341709 ## api/src/test/java/org/apache/iceberg/TestSchema.java: ## @@ -21,29 +21,30 @@ import static org.assertj.core.api.Assertions.assertThatCode; import static o

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826345609 ## api/src/main/java/org/apache/iceberg/DeleteFile.java: ## @@ -31,4 +31,15 @@ public interface DeleteFile extends ContentFile { default List splitOffsets

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on PR #11444: URL: https://github.com/apache/iceberg/pull/11444#issuecomment-2452666774 Test results look like ![image](https://github.com/user-attachments/assets/dc47119c-49c0-4733-a884-09fec1294085) -- This is an automated message from the Apache Git

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11444: URL: https://github.com/apache/iceberg/pull/11444#discussion_r1826341442 ## api/src/test/java/org/apache/iceberg/TestSchema.java: ## @@ -21,29 +21,30 @@ import static org.assertj.core.api.Assertions.assertThatCode; import static o

Re: [PR] REST: AuthManager API [iceberg]

2024-11-01 Thread via GitHub
danielcweeks commented on code in PR #10753: URL: https://github.com/apache/iceberg/pull/10753#discussion_r1826339673 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -378,7 +289,8 @@ public List listTables(SessionContext context, Namespace ns) {

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
rdblue commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826336805 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -448,6 +449,12 @@ public Object get(int pos) { return wrapped.equalityFieldIds(); c

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
rdblue commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826335735 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -339,6 +344,9 @@ protected void internalSet(int pos, T value) { this.sortOrderId = (Integer) va

Re: [PR] feat: more builders and writing manifests [iceberg-go]

2024-11-01 Thread via GitHub
dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1826329602 ## manifest.go: ## @@ -567,6 +584,87 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestList w

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826307104 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -339,6 +344,9 @@ protected void internalSet(int pos, T value) { this.sortOrderId = (Intege

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826304698 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -448,6 +449,12 @@ public Object get(int pos) { return wrapped.equalityFieldIds();

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826304698 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -448,6 +449,12 @@ public Object get(int pos) { return wrapped.equalityFieldIds();

Re: [PR] [WIP] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on code in PR #11443: URL: https://github.com/apache/iceberg/pull/11443#discussion_r1826304698 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -448,6 +449,12 @@ public Object get(int pos) { return wrapped.equalityFieldIds();

[PR] API, Core: Add data file reference to DeleteFile [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi opened a new pull request, #11443: URL: https://github.com/apache/iceberg/pull/11443 This PR adds `referenced_data_file` to our delete metadata as per the design doc for improving position deletes. This PR cannot be merged until the vote is closed. -- This is an automat

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
szehon-ho commented on code in PR #11437: URL: https://github.com/apache/iceberg/pull/11437#discussion_r1826273849 ## core/src/main/java/org/apache/iceberg/util/PropertyUtil.java: ## @@ -57,6 +67,19 @@ public static double propertyAsDouble( return defaultValue; } + pu

[PR] cmd/iceberg: Pass default aws config to glue catalog constructor [iceberg-go]

2024-11-01 Thread via GitHub
glkz opened a new pull request, #187: URL: https://github.com/apache/iceberg-go/pull/187 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826243134 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826235415 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826240929 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826235415 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826235415 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826235415 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826235415 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826215525 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,8 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] Core: Add cardinality to PositionDeleteIndex [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on code in PR #11442: URL: https://github.com/apache/iceberg/pull/11442#discussion_r1826223534 ## core/src/main/java/org/apache/iceberg/deletes/BitmapPositionDeleteIndex.java: ## @@ -87,4 +87,9 @@ public void forEach(LongConsumer consumer) { public Colle

[PR] Core: Add cardinality to PositionDeleteIndex [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi opened a new pull request, #11442: URL: https://github.com/apache/iceberg/pull/11442 This PR adds `cardinality` to `PositionDeleteIndex`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826215525 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,8 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
dramaticlly commented on code in PR #11437: URL: https://github.com/apache/iceberg/pull/11437#discussion_r1826214271 ## core/src/main/java/org/apache/iceberg/util/PropertyUtil.java: ## @@ -57,6 +67,19 @@ public static double propertyAsDouble( return defaultValue; } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826211841 ## api/src/main/java/org/apache/iceberg/types/Type.java: ## @@ -92,6 +93,10 @@ default boolean isListType() { return false; } + default boolean isVa

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826210225 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,8 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826209245 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,8 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] Core: Use RoaringPositionBitmap in position index [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi merged PR #11441: URL: https://github.com/apache/iceberg/pull/11441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Core: Use RoaringPositionBitmap in position index [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi commented on PR #11441: URL: https://github.com/apache/iceberg/pull/11441#issuecomment-2452453644 Thank you, @amogh-jahagirdar! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826206182 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -562,7 +563,7 @@ private static String sanitize(Literal literal, long now, int

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1826205708 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -562,7 +563,7 @@ private static String sanitize(Literal literal, long now, int

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826190048 ## format/spec.md: ## @@ -444,6 +449,9 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associa

Re: [I] optimize `_combine_positional_deletes` [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu commented on issue #1271: URL: https://github.com/apache/iceberg-python/issues/1271#issuecomment-2452407403 @corleyma thats awesome, thanks! Would you like to open a PR and contribute the change? -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826184241 ## format/spec.md: ## @@ -1462,6 +1484,7 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** | **`JSON o

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826183315 ## format/spec.md: ## @@ -1436,6 +1457,7 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** |

Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]

2024-11-01 Thread via GitHub
sungwy commented on issue #1284: URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2452415487 Hey @kevinjqliu thanks for the support! Glad to hear we are on the same page regarding this issue. As with all API changes, I think we would want to deliberate on these

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826183315 ## format/spec.md: ## @@ -1436,6 +1457,7 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** |

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826181717 ## format/spec.md: ## @@ -1287,6 +1307,7 @@ Types are serialized according to this table: |**`struct`**|`JSON object: {`  `"type": "struct",`  `"fields": [ {`

Re: [PR] Deprecate for 0.8.0 release [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu merged PR #1269: URL: https://github.com/apache/iceberg-python/pull/1269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826179192 ## format/spec.md: ## @@ -1297,54 +1308,56 @@ Example This serialization scheme is for storing single values as individual binary values in the lower and u

Re: [PR] Remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu commented on PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#issuecomment-2452403025 Thanks @HonahX for the fix and @Fokko for the review! One step closer to 0.8.0! -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] [🐞] Collection of a few bugs [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu closed issue #864: [🐞] Collection of a few bugs URL: https://github.com/apache/iceberg-python/issues/864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu merged PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu commented on issue #1284: URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2452399942 Thanks for the writeup! >Keep the field_ids when using pyiceberg.schema.Schema, but generate new ones when using pyarrow.Schema +1 to this Here's my t

Re: [PR] Allow union of `{int,long}`, `{float,double}`, etc [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu commented on code in PR #1283: URL: https://github.com/apache/iceberg-python/pull/1283#discussion_r1826131475 ## tests/test_schema.py: ## @@ -1200,11 +1200,33 @@ def test_promote_float_to_double() -> None: assert isinstance(applied.fields[0].field_type, DoubleTyp

Re: [PR] feat: more builders and writing manifests [iceberg-go]

2024-11-01 Thread via GitHub
zeroshade commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1826113696 ## manifest.go: ## @@ -567,6 +584,87 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestList wri

Re: [PR] Pass table-token to commit endpoint [iceberg-python]

2024-11-01 Thread via GitHub
kevinjqliu commented on code in PR #1278: URL: https://github.com/apache/iceberg-python/pull/1278#discussion_r1826115316 ## pyiceberg/catalog/rest.py: ## @@ -777,9 +778,15 @@ def commit_table( identifier = self._identifier_to_tuple_without_catalog(table.identifier)

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
szehon-ho commented on code in PR #11437: URL: https://github.com/apache/iceberg/pull/11437#discussion_r1826111628 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -486,6 +489,10 @@ public int propertyAsInt(String property, int defaultValue) { return Prop

Re: [PR] Core: Add validation for table commit properties [iceberg]

2024-11-01 Thread via GitHub
szehon-ho commented on code in PR #11437: URL: https://github.com/apache/iceberg/pull/11437#discussion_r1826110743 ## core/src/main/java/org/apache/iceberg/util/PropertyUtil.java: ## @@ -100,6 +123,28 @@ public static String propertyAsString( return defaultValue; } +

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
RussellSpitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826053379 ## format/spec.md: ## @@ -1110,6 +1125,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`struct`**|`record`|| |**`li

Re: [PR] Spec: add variant type [iceberg]

2024-11-01 Thread via GitHub
sfc-gh-rspitzer commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1826050630 ## format/spec.md: ## @@ -1110,6 +1125,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`struct`**|`record`|| |**`l

[PR] Core: Use RoaringPositionBitmap in position index [iceberg]

2024-11-01 Thread via GitHub
aokolnychyi opened a new pull request, #11441: URL: https://github.com/apache/iceberg/pull/11441 This PR migrates `BitmapPositionDeleteIndex` to use `RoaringPositionBitmap` in preparation for DVs. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] API: Add Variant data type [iceberg]

2024-11-01 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1825983981 ## api/src/test/java/org/apache/iceberg/TestAccessors.java: ## @@ -247,4 +252,70 @@ public void testEmptySchema() { Schema emptySchema = new Schema(); asser

Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]

2024-11-01 Thread via GitHub
liko9 commented on issue #10127: URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2452011720 Should we reopen this issue since it [was reverted](https://github.com/apache/iceberg/pull/11344) and still exists as a problem? -- This is an automated message from the Apache Gi

Re: [I] Flink/Azure job graph serialization fails when used with storage account shared key authentication [iceberg]

2024-11-01 Thread via GitHub
ms closed issue #10245: Flink/Azure job graph serialization fails when used with storage account shared key authentication URL: https://github.com/apache/iceberg/issues/10245 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] feat: more builders and writing manifests [iceberg-go]

2024-11-01 Thread via GitHub
zeroshade commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1825908854 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2 w

Re: [PR] feat: Safer PartitionSpec & SchemalessPartitionSpec [iceberg-rust]

2024-11-01 Thread via GitHub
c-thiel commented on code in PR #645: URL: https://github.com/apache/iceberg-rust/pull/645#discussion_r1825848112 ## crates/iceberg/src/spec/partition.rs: ## @@ -131,50 +205,22 @@ impl PartitionSpec { /// * Field names /// * Source column ids /// * Transforms -

Re: [I] Serialization of the org.apache.iceberg.io.WriteResult class. [iceberg]

2024-11-01 Thread via GitHub
simonykq commented on issue #10710: URL: https://github.com/apache/iceberg/issues/10710#issuecomment-2451893935 @pvary Thanks for the info. Very informative. I am more of less thinking of letting flink runtime treat it as a Pojo types using `PojoTypeInfo`. Something like this:

Re: [I] Implement `TableProviderFactory` for a `IcebergTableFactory` [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 closed issue #586: Implement `TableProviderFactory` for a `IcebergTableFactory` URL: https://github.com/apache/iceberg-rust/issues/586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] feat: implement IcebergTableProviderFactory for datafusion [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 merged PR #600: URL: https://github.com/apache/iceberg-rust/pull/600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: implement IcebergTableProviderFactory for datafusion [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 commented on PR #600: URL: https://github.com/apache/iceberg-rust/pull/600#issuecomment-2451839743 Thanks @yukkit for this pr! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Flink/Azure job graph serialization fails when used with storage account shared key authentication [iceberg]

2024-11-01 Thread via GitHub
ms commented on issue #10245: URL: https://github.com/apache/iceberg/issues/10245#issuecomment-2451838473 Fixed in 1.6.0 by #10045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-11-01 Thread via GitHub
Fokko commented on code in PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#discussion_r1825762462 ## pyiceberg/table/update/__init__.py: ## @@ -90,7 +90,13 @@ class AddSchemaUpdate(IcebergBaseModel): # This field is required: https://github.com/apache/iceb

Re: [PR] feat: Safer PartitionSpec & SchemalessPartitionSpec [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 commented on PR #645: URL: https://github.com/apache/iceberg-rust/pull/645#issuecomment-2451776760 Also please merge with main branch to keep it updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] feat: Safer PartitionSpec & SchemalessPartitionSpec [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 commented on code in PR #645: URL: https://github.com/apache/iceberg-rust/pull/645#discussion_r1825726996 ## crates/iceberg/src/spec/partition.rs: ## @@ -235,6 +281,21 @@ impl UnboundPartitionSpec { } } +fn has_sequential_ids(field_ids: &[i32]) -> bool { R

Re: [I] iceberg-spark-runtime: Sources jar does not contain any sources [iceberg]

2024-11-01 Thread via GitHub
SauronShepherd commented on issue #11418: URL: https://github.com/apache/iceberg/issues/11418#issuecomment-2451754017 I see. Shouldn't this "spark-runtime" jar be really named "iceberg-spark-all" or "iceberg-spark-with-dependencies"? And, if it's intended for runtime ... why are the

[PR] Allow union of `{int,long}`, `{float,double}`, etc [iceberg-python]

2024-11-01 Thread via GitHub
Fokko opened a new pull request, #1283: URL: https://github.com/apache/iceberg-python/pull/1283 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] fix(type): fix type promote to ignore field name. [iceberg-rust]

2024-11-01 Thread via GitHub
liurenjie1024 merged PR #685: URL: https://github.com/apache/iceberg-rust/pull/685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic