[GitHub] [iceberg-docs] quantsegu opened a new pull request, #192: Adjusted on comments for adding IOMETE as a vendor

2022-12-24 Thread GitBox
quantsegu opened a new pull request, #192: URL: https://github.com/apache/iceberg-docs/pull/192 The small textual adjustments are made. Can you please finalize this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5984: Core, API: Support incremental scanning with branch

2022-12-24 Thread GitBox
hililiwei commented on code in PR #5984: URL: https://github.com/apache/iceberg/pull/5984#discussion_r1056798443 ## api/src/main/java/org/apache/iceberg/IncrementalScan.java: ## @@ -21,6 +21,23 @@ /** API for configuring an incremental scan. */ public interface IncrementalScan

[GitHub] [iceberg] hililiwei commented on pull request #6253: Flink: Write watermark to the snapshot summary

2022-12-24 Thread GitBox
hililiwei commented on PR #6253: URL: https://github.com/apache/iceberg/pull/6253#issuecomment-1364513120 >should we write the metadata as snapshot summary or table properties? It is minimal change to write as snapshot summary as shown in this PR or PR https://github.com/apache/iceberg/pull

[GitHub] [iceberg] hililiwei commented on pull request #6160: Flink: Support locality with LocalitySplitAssigner

2022-12-24 Thread GitBox
hililiwei commented on PR #6160: URL: https://github.com/apache/iceberg/pull/6160#issuecomment-1364514213 @stevenzwu @openinx @rdblue @Fokko @pvary could you please take a look at it when you get a chance? thx. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [iceberg-docs] RussellSpitzer merged pull request #192: Adjusted on comments for adding IOMETE as a vendor

2022-12-24 Thread GitBox
RussellSpitzer merged PR #192: URL: https://github.com/apache/iceberg-docs/pull/192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

[GitHub] [iceberg] ConeyLiu commented on pull request #3249: Optimized spark vectorized read parquet decimal

2022-12-24 Thread GitBox
ConeyLiu commented on PR #3249: URL: https://github.com/apache/iceberg/pull/3249#issuecomment-1364527752 @nastra just addressed the comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852402 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/StructInternalRow.java: ## @@ -356,4 +357,23 @@ private GenericArrayData fillArray( return

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852402 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/StructInternalRow.java: ## @@ -356,4 +357,23 @@ private GenericArrayData fillArray( return

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852535 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -147,9 +152,9 @@ protected Statistics estimateStatistics(Snapshot snapshot)

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852815 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -200,6 +200,16 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { /**

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852878 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -554,6 +554,10 @@ public List fields() { return lazyFieldList(); } +public boolean

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852903 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java: ## @@ -140,4 +144,35 @@ public synchronized void testU

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852915 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -42,4 +42,9 @@ private SparkSQLProperties() {} // Controls whether to c

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056852950 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java: ## @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056862209 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java: ## @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on code in PR #6371: URL: https://github.com/apache/iceberg/pull/6371#discussion_r1056862319 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java: ## @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] aokolnychyi merged pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi merged PR #6371: URL: https://github.com/apache/iceberg/pull/6371 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[GitHub] [iceberg] aokolnychyi commented on pull request #6371: Spark 3.3: Support storage-partitioned joins

2022-12-24 Thread GitBox
aokolnychyi commented on PR #6371: URL: https://github.com/apache/iceberg/pull/6371#issuecomment-1364570187 Thanks for reviewing, @RussellSpitzer @sunchao @zinking @rdblue! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [iceberg] aokolnychyi commented on issue #430: Support bucket table for Iceberg

2022-12-24 Thread GitBox
aokolnychyi commented on issue #430: URL: https://github.com/apache/iceberg/issues/430#issuecomment-1364570547 I am excited to announce that support for storage-partitioned joins has been merged into master. It will be shipped in 1.2.0. Thanks everyone involved, especially @sunchao. I am

[GitHub] [iceberg] aokolnychyi closed issue #430: Support bucket table for Iceberg

2022-12-24 Thread GitBox
aokolnychyi closed issue #430: Support bucket table for Iceberg URL: https://github.com/apache/iceberg/issues/430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056874376 ## python/pyiceberg/io/pyarrow.py: ## @@ -437,3 +465,198 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p def expression_to_pyarrow

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056874464 ## python/pyiceberg/io/pyarrow.py: ## @@ -437,3 +465,198 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p def expression_to_pyarrow

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056874748 ## python/pyiceberg/io/pyarrow.py: ## @@ -437,3 +465,198 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p def expression_to_pyarrow

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056875584 ## python/pyiceberg/schema.py: ## @@ -1046,3 +1055,79 @@ def _project_map(map_type: MapType, value_result: IcebergType) -> MapType: value_type=value_res

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056875610 ## python/pyiceberg/schema.py: ## @@ -1046,3 +1055,79 @@ def _project_map(map_type: MapType, value_result: IcebergType) -> MapType: value_type=value_res

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056875777 ## python/pyiceberg/exceptions.py: ## @@ -86,3 +86,7 @@ class NotInstalledError(Exception): class SignError(Exception): """Raises when unable to sign a S3 request

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056875820 ## python/pyiceberg/expressions/visitors.py: ## @@ -753,3 +756,89 @@ def inclusive_projection( schema: Schema, spec: PartitionSpec, case_sensitive: bool = True ) ->

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056875859 ## python/pyiceberg/expressions/visitors.py: ## @@ -753,3 +756,89 @@ def inclusive_projection( schema: Schema, spec: PartitionSpec, case_sensitive: bool = True ) ->

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056876255 ## python/pyiceberg/types.py: ## @@ -268,6 +268,10 @@ def __init__(self, *fields: NestedField, **data: Any): data["fields"] = fields super().__init_

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056876528 ## python/tests/avro/test_resolver.py: ## @@ -164,17 +163,17 @@ def test_resolver_change_type() -> None: def test_promote_int_to_long() -> None: -assert promote(

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056877937 ## python/pyiceberg/schema.py: ## @@ -1046,3 +1055,79 @@ def _project_map(map_type: MapType, value_result: IcebergType) -> MapType: value_type=value_res

[GitHub] [iceberg] Fokko commented on a diff in pull request #6437: Python: Projection by Field ID

2022-12-24 Thread GitBox
Fokko commented on code in PR #6437: URL: https://github.com/apache/iceberg/pull/6437#discussion_r1056878066 ## python/pyiceberg/schema.py: ## @@ -1046,3 +1055,79 @@ def _project_map(map_type: MapType, value_result: IcebergType) -> MapType: value_type=value_res

[GitHub] [iceberg] github-actions[bot] commented on issue #5139: Historical time travel imports

2022-12-24 Thread GitBox
github-actions[bot] commented on issue #5139: URL: https://github.com/apache/iceberg/issues/5139#issuecomment-1364598871 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] commented on issue #5141: No way to rollback first commit in table

2022-12-24 Thread GitBox
github-actions[bot] commented on issue #5141: URL: https://github.com/apache/iceberg/issues/5141#issuecomment-1364598864 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: WIP: Delta, Spark: Adding support for Migrating Delta Lake Table to Iceberg Table

2022-12-24 Thread GitBox
JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1056753664 ## data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java: ## @@ -161,7 +161,7 @@ private static Metrics getAvroMetrics(Path path, Configuration conf) {

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6491: Build: Bump actions/stale from 6.0.1 to 7.0.0

2022-12-24 Thread GitBox
dependabot[bot] opened a new pull request, #6491: URL: https://github.com/apache/iceberg/pull/6491 Bumps [actions/stale](https://github.com/actions/stale) from 6.0.1 to 7.0.0. Release notes Sourced from https://github.com/actions/stale/releases";>actions/stale's releases. v7

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6492: Build: Bump adlfs from 2022.10.0 to 2022.11.2 in /python

2022-12-24 Thread GitBox
dependabot[bot] opened a new pull request, #6492: URL: https://github.com/apache/iceberg/pull/6492 Bumps [adlfs](https://github.com/dask/adlfs) from 2022.10.0 to 2022.11.2. Changelog Sourced from https://github.com/fsspec/adlfs/blob/main/CHANGELOG.md";>adlfs's changelog. 202

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6493: Build: Bump coverage from 6.5.0 to 7.0.1 in /python

2022-12-24 Thread GitBox
dependabot[bot] opened a new pull request, #6493: URL: https://github.com/apache/iceberg/pull/6493 Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.5.0 to 7.0.1. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's changelog.

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6494: Build: Bump moto from 4.0.11 to 4.0.12 in /python

2022-12-24 Thread GitBox
dependabot[bot] opened a new pull request, #6494: URL: https://github.com/apache/iceberg/pull/6494 Bumps [moto](https://github.com/spulec/moto) from 4.0.11 to 4.0.12. Changelog Sourced from https://github.com/spulec/moto/blob/master/CHANGELOG.md";>moto's changelog. 4.0.12

[GitHub] [iceberg] nickvazz commented on issue #6061: [Python] Add examples

2022-12-24 Thread GitBox
nickvazz commented on issue #6061: URL: https://github.com/apache/iceberg/issues/6061#issuecomment-1364628562 Would love to see a minimal getting started / setup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL