[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6627: Docs: Update spark SQL examples for time travel to branches and tags

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #6627: URL: https://github.com/apache/iceberg/pull/6627#discussion_r1082918805 ## docs/spark-queries.md: ## @@ -95,21 +95,37 @@ The above list is in order of priority. For example: a matching catalog will tak SQL -Spark 3.3 and

[GitHub] [iceberg] jackye1995 commented on pull request #6627: Docs: Update spark SQL examples for time travel to branches and tags

2023-01-20 Thread via GitHub
jackye1995 commented on PR #6627: URL: https://github.com/apache/iceberg/pull/6627#issuecomment-1398780784 Thanks for the update and reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [iceberg] jackye1995 merged pull request #6627: Docs: Update spark SQL examples for time travel to branches and tags

2023-01-20 Thread via GitHub
jackye1995 merged PR #6627: URL: https://github.com/apache/iceberg/pull/6627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[GitHub] [iceberg] jackye1995 commented on pull request #6586: AWS: make warehouse path optional for read only catalog use cases

2023-01-20 Thread via GitHub
jackye1995 commented on PR #6586: URL: https://github.com/apache/iceberg/pull/6586#issuecomment-1398781852 Thanks for testing with Glue @aajisaka ! And thanks for the review @amogh-jahagirdar @nastra -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [iceberg] jackye1995 merged pull request #6586: AWS: make warehouse path optional for read only catalog use cases

2023-01-20 Thread via GitHub
jackye1995 merged PR #6586: URL: https://github.com/apache/iceberg/pull/6586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[GitHub] [iceberg] szehon-ho commented on issue #6257: Partitions metadata table shows old partitions

2023-01-20 Thread via GitHub
szehon-ho commented on issue #6257: URL: https://github.com/apache/iceberg/issues/6257#issuecomment-1398785459 Yea I admit that is annoying. Maybe just the fact to add delete_files column will help know that perhaps the record_count may change? (As well as documenting of course). But agr

[GitHub] [iceberg] yyanyy commented on a diff in pull request #6627: Docs: Update spark SQL examples for time travel to branches and tags

2023-01-20 Thread via GitHub
yyanyy commented on code in PR #6627: URL: https://github.com/apache/iceberg/pull/6627#discussion_r1082937202 ## docs/spark-queries.md: ## @@ -95,21 +95,37 @@ The above list is in order of priority. For example: a matching catalog will tak SQL -Spark 3.3 and later sup

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6591: Core: Avoid creating new metadata file when `registerTable` API is used

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6591: URL: https://github.com/apache/iceberg/pull/6591#discussion_r1082932748 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -154,6 +154,12 @@ protected void disableRefresh() { this.shouldRefresh = false;

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6410: Configurable metrics reporter by catalog properties

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6410: URL: https://github.com/apache/iceberg/pull/6410#discussion_r1082942925 ## core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java: ## @@ -301,4 +305,16 @@ protected static String fullTableName(String catalogName, TableIdentifier

[GitHub] [iceberg] szehon-ho commented on pull request #6410: Configurable metrics reporter by catalog properties

2023-01-20 Thread via GitHub
szehon-ho commented on PR #6410: URL: https://github.com/apache/iceberg/pull/6410#issuecomment-1398807289 I think its fine with me if we can fix the failures -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [iceberg] yyanyy commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-20 Thread via GitHub
yyanyy commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1082942731 ## core/src/main/java/org/apache/iceberg/view/SQLViewRepresentationParser.java: ## @@ -0,0 +1,119 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-20 Thread via GitHub
singhpk234 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1082969077 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,370 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-20 Thread via GitHub
singhpk234 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1082969487 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,370 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-20 Thread via GitHub
singhpk234 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1082969487 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,370 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-20 Thread via GitHub
jackye1995 commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1082987146 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,370 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] [iceberg] aokolnychyi opened a new pull request, #6633: Spark 3.3: Fix predicate pushdown for copy-on-write MERGE commands

2023-01-20 Thread via GitHub
aokolnychyi opened a new pull request, #6633: URL: https://github.com/apache/iceberg/pull/6633 This PR fixes predicate pushdown for copy-on-write MERGE commands, which was broken after #6534. This change contains a test that would previously fail and lead to a data correctness issue. --

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6633: Spark 3.3: Fix predicate pushdown for copy-on-write MERGE commands

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6633: URL: https://github.com/apache/iceberg/pull/6633#discussion_r1082998249 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -187,14 +187,12 @@ object RewriteMergeIntoTa

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6633: Spark 3.3: Fix predicate pushdown for copy-on-write MERGE commands

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6633: URL: https://github.com/apache/iceberg/pull/6633#discussion_r1082998249 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -187,14 +187,12 @@ object RewriteMergeIntoTa

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

2023-01-20 Thread via GitHub
amogh-jahagirdar opened a new pull request, #6634: URL: https://github.com/apache/iceberg/pull/6634 Fix for https://github.com/apache/iceberg/issues/6632 Moving to draft as I read more of the code to make sure this handles different failure cases properly and will add tests if determi

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083023727 ## api/src/main/java/org/apache/iceberg/SnapshotUpdate.java: ## @@ -71,4 +71,8 @@ default ThisT toBranch(String branch) { "Cannot commit to branch

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1059510877 ## delta-lake/src/main/java/org/apache/iceberg/delta/SupportMigrationFromDeltaLake.java: ## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083026645 ## core/src/main/java/org/apache/iceberg/BasePositionDeletesScanTask.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083065040 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083065272 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083065455 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083065602 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,372 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083068421 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083080152 ## core/src/main/java/org/apache/iceberg/SnapshotScan.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083083501 ## core/src/main/java/org/apache/iceberg/BaseMetadataTable.java: ## @@ -64,9 +64,12 @@ protected BaseMetadataTable(TableOperations ops, Table table, String name) {

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083087907 ## core/src/main/java/org/apache/iceberg/BaseMetadataTable.java: ## @@ -73,9 +73,12 @@ protected BaseMetadataTable(Table table, String name) { */ static Part

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083088187 ## core/src/main/java/org/apache/iceberg/BaseMetadataTable.java: ## @@ -64,9 +64,12 @@ protected BaseMetadataTable(TableOperations ops, Table table, String name) {

[GitHub] [iceberg] stevenzwu merged pull request #6631: Flink: backport PR #6584 to 1.14 and 1.15 for Avro GenericRecord in FLIP-27 source

2023-01-20 Thread via GitHub
stevenzwu merged PR #6631: URL: https://github.com/apache/iceberg/pull/6631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg] jackye1995 commented on pull request #6617: Spark: Spark SQL Extensions for create branch

2023-01-20 Thread via GitHub
jackye1995 commented on PR #6617: URL: https://github.com/apache/iceberg/pull/6617#issuecomment-1399009614 I think this PR is mostly ready to go. I see there is a comment in design doc from @flyrain: ``` "VERSION" is used in Iceberg to indicate any table changes including table pr

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on PR #6624: URL: https://github.com/apache/iceberg/pull/6624#issuecomment-1399009685 @kingeasternsun A maintainer should take a look when they get a chance. @RussellSpitzer @aokolnychyi @szehon-ho @jackye1995 when you get a chance could you take a look? Thanks!

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083099208 ## core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java: ## @@ -79,23 +79,32 @@ public ReplacePartitions validateNoConflictingData() { return this; }

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083099568 ## core/src/main/java/org/apache/iceberg/BaseRowDelta.java: ## @@ -96,23 +97,37 @@ public RowDelta validateNoConflictingDeleteFiles() { } @Override - protected

[GitHub] [iceberg] stevenzwu opened a new pull request, #6635: Flink: add table setter to FLIP-27 IcebergSource#Builder.

2023-01-20 Thread via GitHub
stevenzwu opened a new pull request, #6635: URL: https://github.com/apache/iceberg/pull/6635 This is to avoid double loading if table is already loaded before the builder. This is also the same pattern as the pre FLIP-27 FlinkSource#Builder. -- This is an automated message from the Apache

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083101029 ## core/src/main/java/org/apache/iceberg/BaseRowDelta.java: ## @@ -96,23 +97,37 @@ public RowDelta validateNoConflictingDeleteFiles() { } @Override - protected

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083103566 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -164,40 +173,43 @@ public void testOverwriteFailsDelete() { "Should reject commit with file n

[GitHub] [iceberg] jackye1995 opened a new issue, #6636: Unclear messaging about Glue catalog locking

2023-01-20 Thread via GitHub
jackye1995 opened a new issue, #6636: URL: https://github.com/apache/iceberg/issues/6636 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 Based on reader feedback, the message presented in https://iceberg.apache.org/docs/l

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083103875 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -164,40 +173,43 @@ public void testOverwriteFailsDelete() { "Should reject commit with file n

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083109459 ## core/src/test/java/org/apache/iceberg/TestReplacePartitions.java: ## @@ -114,20 +122,22 @@ public void testReplaceAndMergeOnePartition() { // ensure the overwrit

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083112330 ## core/src/test/java/org/apache/iceberg/TestRowDelta.java: ## @@ -39,18 +39,32 @@ import org.apache.iceberg.relocated.com.google.common.collect.Sets; import org.junit

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083113239 ## core/src/test/java/org/apache/iceberg/TestRowDelta.java: ## @@ -81,155 +95,171 @@ public void testAddDeleteFile() { @Test public void testValidateDataFilesExi

[GitHub] [iceberg] aokolnychyi commented on pull request #6012: Spark 3.3: Add a procedure to generate table changes

2023-01-20 Thread via GitHub
aokolnychyi commented on PR #6012: URL: https://github.com/apache/iceberg/pull/6012#issuecomment-1399044913 Getting to this PR soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083116981 ## core/src/main/java/org/apache/iceberg/SnapshotScan.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083117144 ## core/src/main/java/org/apache/iceberg/util/PartitionUtil.java: ## @@ -91,7 +91,7 @@ private PartitionUtil() {} } // adapts the provided partition data to m

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083117287 ## core/src/main/java/org/apache/iceberg/SnapshotScan.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083117457 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083117603 ## core/src/main/java/org/apache/iceberg/BasePositionDeletesScanTask.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083117789 ## core/src/test/java/org/apache/iceberg/TestRowDelta.java: ## @@ -81,155 +95,171 @@ public void testAddDeleteFile() { @Test public void testValidateDataFilesExi

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083118880 ## core/src/test/java/org/apache/iceberg/TestRowDelta.java: ## @@ -81,155 +95,171 @@ public void testAddDeleteFile() { @Test public void testValidateDa

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-20 Thread via GitHub
jackye1995 commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083145098 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationExcep

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-20 Thread via GitHub
jackye1995 commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083145300 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationExcep

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083191910 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -397,8 +405,10 @@ protected void validateNoNewDeletesForDataFiles( TableMetadata base

[GitHub] [iceberg] rdblue commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
rdblue commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083195038 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -460,8 +471,8 @@ private void validateNoNewDeletesForDataFiles( * @param dataFilter an e

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083230533 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-20 Thread via GitHub
amogh-jahagirdar commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1399167754 Thanks for the reviews @rdblue ! @namrathamyske I raised a PR to your branch for deprecating the old validation methods and updating rev API https://github.com/namrathamyske/iceb

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-20 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083237847 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationE

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-20 Thread via GitHub
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1083250975 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1040,4 +1047,195 @@ public void testAllManifestsTableSnapshotNot() { expectedMan

[GitHub] [iceberg] hililiwei opened a new pull request, #6637: Spark: Spark SQL Extensions for create tag

2023-01-20 Thread via GitHub
hililiwei opened a new pull request, #6637: URL: https://github.com/apache/iceberg/pull/6637 Co-authored-by: Amogh Jahagirdar Co-authored-by: chidayong <247070...@qq.com> ## What is the purpose of the change Implement the syntax in the following documents: https://docs.googl

[GitHub] [iceberg] hililiwei commented on pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-20 Thread via GitHub
hililiwei commented on PR #6637: URL: https://github.com/apache/iceberg/pull/6637#issuecomment-1399194014 In the original proposal, it was "[RETAIN For interval {DAYS | HOURS | MINUTES}]", but in keeping with CREATE BRANCH, I removed the" For" key. What do you think about that? @jackye1995

[GitHub] [iceberg] jackye1995 commented on pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-20 Thread via GitHub
jackye1995 commented on PR #6637: URL: https://github.com/apache/iceberg/pull/6637#issuecomment-1399195757 Good point, +1 for only RETAIN because https://docs.databricks.com/sql/language-manual/delta-vacuum.html -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [iceberg] jackye1995 commented on pull request #6617: Spark: Spark SQL Extensions for create branch

2023-01-21 Thread via GitHub
jackye1995 commented on PR #6617: URL: https://github.com/apache/iceberg/pull/6617#issuecomment-1399207001 Looks like there are quite a few duplicates for #6637 without merging this one. In that case given there are 2 committer votes, I will first merge this one to unblock that PR, and will

[GitHub] [iceberg] jackye1995 merged pull request #6617: Spark: Spark SQL Extensions for create branch

2023-01-21 Thread via GitHub
jackye1995 merged PR #6617: URL: https://github.com/apache/iceberg/pull/6617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-21 Thread via GitHub
amogh-jahagirdar commented on PR #6637: URL: https://github.com/apache/iceberg/pull/6637#issuecomment-1399286553 > In the original proposal, it was "[RETAIN For interval {DAYS | HOURS | MINUTES}]", but in keeping with CREATE BRANCH, I removed the" For" key. What do you think about that? @ja

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

2023-01-21 Thread via GitHub
amogh-jahagirdar commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083324041 ## core/src/test/java/org/apache/iceberg/TestTransaction.java: ## @@ -771,4 +771,14 @@ public void testSimpleTransactionNotDeletingMetadataOnUnknownSate() thr

[GitHub] [iceberg] Fokko merged pull request #6607: Python: Fix `test_missing_uri` unit test

2023-01-21 Thread via GitHub
Fokko merged PR #6607: URL: https://github.com/apache/iceberg/pull/6607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083329928 ## core/src/test/java/org/apache/iceberg/TestTransaction.java: ## @@ -771,4 +771,14 @@ public void testSimpleTransactionNotDeletingMetadataOnUnknownSate() throws IO

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6637: URL: https://github.com/apache/iceberg/pull/6637#discussion_r1083330622 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -168,34 +169,61 @@ fieldList ; n

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083331517 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationExcep

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083331517 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationExcep

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083331843 ## api/src/main/java/org/apache/iceberg/SnapshotUpdate.java: ## @@ -71,4 +71,8 @@ default ThisT toBranch(String branch) { "Cannot commit to branch %s: %

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-21 Thread via GitHub
amogh-jahagirdar commented on code in PR #6637: URL: https://github.com/apache/iceberg/pull/6637#discussion_r1083371736 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -73,6 +73,7 @@ statement | AL

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-21 Thread via GitHub
amogh-jahagirdar opened a new pull request, #6638: URL: https://github.com/apache/iceberg/pull/6638 Co-authored-by: liliwei liliwe...@huawei.com Co-authored-by: xuwei xuwei...@huawei.com Co-authored-by: chidayong chidayo...@h-partners.com CC: @jackye1995 @hililiwei @flyrain --

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-21 Thread via GitHub
amogh-jahagirdar commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1083379723 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | AL

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-21 Thread via GitHub
amogh-jahagirdar commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1083379803 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestReplaceBranch.java: ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache So

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-21 Thread via GitHub
jackye1995 commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1083380371 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -74,6 +74,7 @@ statement | ALTER TA

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6637: Spark: Spark SQL Extensions for create tag

2023-01-21 Thread via GitHub
hililiwei commented on code in PR #6637: URL: https://github.com/apache/iceberg/pull/6637#discussion_r1083385463 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -73,6 +73,7 @@ statement | ALTER TAB

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-21 Thread via GitHub
namrathamyske commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083386045 ## core/src/main/java/org/apache/iceberg/BaseRowDelta.java: ## @@ -96,23 +97,37 @@ public RowDelta validateNoConflictingDeleteFiles() { } @Override - pro

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-21 Thread via GitHub
namrathamyske commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1083386045 ## core/src/main/java/org/apache/iceberg/BaseRowDelta.java: ## @@ -96,23 +97,37 @@ public RowDelta validateNoConflictingDeleteFiles() { } @Override - pro

[GitHub] [iceberg] JonasJ-ap opened a new pull request, #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table

2023-01-21 Thread via GitHub
JonasJ-ap opened a new pull request, #6642: URL: https://github.com/apache/iceberg/pull/6642 This PR is under construction, but I want to put it here for some initial feedback and discussion about the conversion from Apache Hudi to Apache Iceberg ## Overview This PR aims to add a m

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-22 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083420820 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/SnapshotTableProcedure.java: ## @@ -93,10 +94,20 @@ public InternalRow[] call(InternalRow a

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-22 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083420860 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/MigrateTableProcedure.java: ## @@ -99,7 +100,13 @@ public InternalRow[] call(InternalRow ar

[GitHub] [iceberg] Fokko merged pull request #6641: Build: Bump rich from 13.1.0 to 13.2.0 in /python

2023-01-22 Thread via GitHub
Fokko merged PR #6641: URL: https://github.com/apache/iceberg/pull/6641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python

2023-01-22 Thread via GitHub
Fokko commented on PR #6640: URL: https://github.com/apache/iceberg/pull/6640#issuecomment-1399445639 Superseded by https://github.com/apache/iceberg/pull/6643 which bumps all the fsspec packages in sync -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [iceberg] Fokko closed pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python

2023-01-22 Thread via GitHub
Fokko closed pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python URL: https://github.com/apache/iceberg/pull/6640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] Fokko merged pull request #6639: Build: Bump pandas from 1.5.2 to 1.5.3 in /python

2023-01-22 Thread via GitHub
Fokko merged PR #6639: URL: https://github.com/apache/iceberg/pull/6639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-22 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083428466 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationE

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-22 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083440870 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/MigrateTableProcedure.java: ## @@ -39,7 +39,8 @@ class MigrateTableProcedure extends BasePr

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

2023-01-22 Thread via GitHub
kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083441291 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationE

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463683 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463816 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463759 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463953 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Data: java api add GenericTaskWriter and add write demo to Doc.

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083465798 ## data/src/main/java/org/apache/iceberg/data/GenericTaskWriter.java: ## @@ -0,0 +1,79 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Data: java api add GenericTaskWriter and add write demo to Doc.

2023-01-22 Thread via GitHub
youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083466424 ## data/src/main/java/org/apache/iceberg/data/GenericTaskWriter.java: ## @@ -0,0 +1,79 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] youngxinler commented on pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

2023-01-22 Thread via GitHub
youngxinler commented on PR #6554: URL: https://github.com/apache/iceberg/pull/6554#issuecomment-1399507127 @RussellSpitzer Can I trouble you if you have time to do a review? About improve Test Coverage of RowGroupFilter Code with Nans. I have made changes. -- This is an automated messag

[GitHub] [iceberg] Fokko commented on a diff in pull request #6590: Python: Add sql command to the CLI

2023-01-22 Thread via GitHub
Fokko commented on code in PR #6590: URL: https://github.com/apache/iceberg/pull/6590#discussion_r1083538914 ## python/pyiceberg/io/pyarrow.py: ## @@ -470,6 +472,59 @@ def expression_to_pyarrow(expr: BooleanExpression) -> pc.Expression: return boolean_expression_visit(expr

[GitHub] [iceberg] rubenvdg opened a new pull request, #6644: Python: Add support for static table

2023-01-22 Thread via GitHub
rubenvdg opened a new pull request, #6644: URL: https://github.com/apache/iceberg/pull/6644 This PR proposes adding support for static tables (i.e., reading a table directly from a metadata file without using a catalog, see also https://github.com/apache/iceberg/issues/6430). Happy to hear

  1   2   3   4   5   6   7   8   9   10   >