[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1073189489 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -372,6 +378,13 @@ public AuthSession( this.credential = credential; } +// thi

[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1073189146 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -350,6 +350,7 @@ static Long expiresAtMillis(String token) { /** Class to handle authoriza

[GitHub] [iceberg] rbalamohan commented on pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-17 Thread GitBox
rbalamohan commented on PR #6432: URL: https://github.com/apache/iceberg/pull/6432#issuecomment-1386579067 Addressed review comments in latest PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1073155042 ## docs/configuration.md: ## @@ -175,8 +175,13 @@ The HMS table locking is a 2-step process: | iceberg.hive.lock-heartbeat-interval-ms | 24 (4 min) | The heartbea

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1073151680 ## hive-metastore/src/main/java/org/apache/iceberg/hive/NoLock.java: ## @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1073150734 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,533 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
amogh-jahagirdar commented on code in PR #6612: URL: https://github.com/apache/iceberg/pull/6612#discussion_r1073137599 ## format/view-spec.md: ## @@ -107,7 +107,7 @@ This type of representation stores the original view definition in SQL and its S | Required/Optional | Field N

[GitHub] [iceberg] amogh-jahagirdar closed pull request #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
amogh-jahagirdar closed pull request #6612: Spec: View spec rename sql field to query URL: https://github.com/apache/iceberg/pull/6612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [iceberg] RussellSpitzer commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386460001 Your metastore error says that no external hive metastore was configured and it's falling back to local file based hive mode. That's probably not what you want -- This is a

[GitHub] [iceberg] RussellSpitzer commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386459320 That is because released are Flink version specific after that point, latest release is now is 1.1 https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
namrathamyske commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1073043781 ## core/src/test/java/org/apache/iceberg/TableTestBase.java: ## @@ -458,6 +458,18 @@ void validateTableFiles(Table tbl, DataFile... expectedFiles) { Assert.

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386456036 I tried `0.13.2` and got the following error: ``` Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence opera

[GitHub] [iceberg] xuzhiwen1255 commented on a diff in pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

2023-01-17 Thread GitBox
xuzhiwen1255 commented on code in PR #6614: URL: https://github.com/apache/iceberg/pull/6614#discussion_r1073071187 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcTableOperations.java: ## @@ -251,4 +255,14 @@ private Map getTable() return table; }); } +

[GitHub] [iceberg] xuzhiwen1255 commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

2023-01-17 Thread GitBox
xuzhiwen1255 commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1386446054 @stevenzwu Can you have a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [iceberg] xuzhiwen1255 commented on issue #6455: flink streaming query Cannot get a client from a closed pool

2023-01-17 Thread GitBox
xuzhiwen1255 commented on issue #6455: URL: https://github.com/apache/iceberg/issues/6455#issuecomment-1386445704 I found the cause of the problem and tried to solve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg] xuzhiwen1255 opened a new pull request, #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

2023-01-17 Thread GitBox
xuzhiwen1255 opened a new pull request, #6614: URL: https://github.com/apache/iceberg/pull/6614 Solve the problems of #6455 cause : > lazyTable() will open the catalog, then loadTable,loadTable will build tableOps into the loaded table, After the table is loaded, the close method

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386445139 Latest available version of `iceberg-flink-runtime` is `0.12.1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
namrathamyske commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1073044093 ## core/src/test/java/org/apache/iceberg/TableTestBase.java: ## @@ -472,6 +484,20 @@ void validateTableDeleteFiles(Table tbl, DeleteFile... expectedFiles) {

[GitHub] [iceberg] namrathamyske commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
namrathamyske commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1073043781 ## core/src/test/java/org/apache/iceberg/TableTestBase.java: ## @@ -458,6 +458,18 @@ void validateTableFiles(Table tbl, DataFile... expectedFiles) { Assert.

[GitHub] [iceberg] namrathamyske commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
namrathamyske commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1386396949 @rdblue Appreciate your feedback on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1073036477 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1073036477 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
hililiwei commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1073030444 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1073030055 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1073029935 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1073029817 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1073029718 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
szehon-ho commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1073029038 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] RussellSpitzer commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386371990 Have you tried with more recent version? The directory structure really doesn't matter and I'm pretty sure you are just hitting an older version bug. -- This is an automated

[GitHub] [iceberg] yegangy0718 commented on a diff in pull request #6382: Implement ShuffleOperator to collect data statistics

2023-01-17 Thread GitBox
yegangy0718 commented on code in PR #6382: URL: https://github.com/apache/iceberg/pull/6382#discussion_r1073022432 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/ShuffleOperator.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [iceberg] jackye1995 commented on pull request #6611: Spec: Make schema-id required for SQL View Representation

2023-01-17 Thread GitBox
jackye1995 commented on PR #6611: URL: https://github.com/apache/iceberg/pull/6611#issuecomment-1386354696 > I think the reason it was optional is to support cases where views are created via API rather than by a SQL parser Nice that we are clarifying this now. If we support this case

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072999500 ## api/src/main/java/org/apache/iceberg/view/ViewRepresentation.java: ## @@ -18,21 +18,16 @@ */ package org.apache.iceberg.view; -import java.util.Locale; +impo

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386332134 cc/ @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386300159 @RussellSpitzer I forgot that we also partition by another field using `bucket` transform. original spec: ```json "partition-spec" : [ { "name" : "timestamp1_day

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-17 Thread GitBox
huaxingao commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1072976064 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.or

[GitHub] [iceberg] rdblue commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
rdblue commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072972514 ## api/src/main/java/org/apache/iceberg/view/ViewRepresentation.java: ## @@ -18,21 +18,16 @@ */ package org.apache.iceberg.view; -import java.util.Locale; +import o

[GitHub] [iceberg] rdblue commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
rdblue commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072972271 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -18,14 +18,17 @@ */ package org.apache.iceberg.view; +import edu.umd.cs.findbugs.anno

[GitHub] [iceberg] rdblue commented on pull request #6611: Spec: Make schema-id required for SQL View Representation

2023-01-17 Thread GitBox
rdblue commented on PR #6611: URL: https://github.com/apache/iceberg/pull/6611#issuecomment-1386288340 I think the reason it was optional is to support cases where views are created via API rather than by a SQL parser. If you don't have an analyzer, you may not know the output of the SQL qu

[GitHub] [iceberg] huaxingao commented on pull request #6252: push down min/max/count to iceberg

2023-01-17 Thread GitBox
huaxingao commented on PR #6252: URL: https://github.com/apache/iceberg/pull/6252#issuecomment-1386287383 @rdblue > I was hoping that you would take over the PR Will do. Thank you very much for your reply! -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [iceberg] rdblue commented on pull request #6252: push down min/max/count to iceberg

2023-01-17 Thread GitBox
rdblue commented on PR #6252: URL: https://github.com/apache/iceberg/pull/6252#issuecomment-1386283202 @huaxingao, I was hoping that you would take over the PR. I'm sorry that I missed your note saying that you were waiting for me to get it in! Please incorporate those changes into yours.

[GitHub] [iceberg] rdblue commented on pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-17 Thread GitBox
rdblue commented on PR #6432: URL: https://github.com/apache/iceberg/pull/6432#issuecomment-1386282157 Thanks, @rbalamohan! It's close, just a couple more things to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg] rdblue commented on a diff in pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-17 Thread GitBox
rdblue commented on code in PR #6432: URL: https://github.com/apache/iceberg/pull/6432#discussion_r1072967562 ## core/src/main/java/org/apache/iceberg/deletes/Deletes.java: ## @@ -135,6 +141,11 @@ public static StructLikeSet toEqualitySet( } } + @VisibleForTesting +

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072966195 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,372 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] rdblue commented on a diff in pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-17 Thread GitBox
rdblue commented on code in PR #6432: URL: https://github.com/apache/iceberg/pull/6432#discussion_r1072965803 ## core/src/main/java/org/apache/iceberg/SystemProperties.java: ## @@ -41,7 +41,10 @@ private SystemProperties() {} public static final int IO_MANIFEST_CACHE_MAX_FI

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072965261 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] rdblue commented on a diff in pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-17 Thread GitBox
rdblue commented on code in PR #6432: URL: https://github.com/apache/iceberg/pull/6432#discussion_r1072965362 ## core/src/main/java/org/apache/iceberg/SystemProperties.java: ## @@ -41,7 +41,10 @@ private SystemProperties() {} public static final int IO_MANIFEST_CACHE_MAX_FI

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072965261 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] github-actions[bot] closed issue #4930: Schema evolution on migrated Hive tables

2023-01-17 Thread GitBox
github-actions[bot] closed issue #4930: Schema evolution on migrated Hive tables URL: https://github.com/apache/iceberg/issues/4930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[GitHub] [iceberg] github-actions[bot] commented on issue #5058: iceberg v2 table cannot expire delete files after rewrite datafile action

2023-01-17 Thread GitBox
github-actions[bot] commented on issue #5058: URL: https://github.com/apache/iceberg/issues/5058#issuecomment-1386268825 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] commented on issue #4930: Schema evolution on migrated Hive tables

2023-01-17 Thread GitBox
github-actions[bot] commented on issue #4930: URL: https://github.com/apache/iceberg/issues/4930#issuecomment-1386268858 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] closed issue #5058: iceberg v2 table cannot expire delete files after rewrite datafile action

2023-01-17 Thread GitBox
github-actions[bot] closed issue #5058: iceberg v2 table cannot expire delete files after rewrite datafile action URL: https://github.com/apache/iceberg/issues/5058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072956183 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072953759 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072953183 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] rdblue commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
rdblue commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072952176 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -372,6 +378,13 @@ public AuthSession( this.credential = credential; } +// thi

[GitHub] [iceberg] rdblue commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
rdblue commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072951550 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -350,6 +350,7 @@ static Long expiresAtMillis(String token) { /** Class to handle authoriza

[GitHub] [iceberg] RussellSpitzer commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386250259 That is valid, it basically represents a transform that previously existed but was dropped. This is how V1 tables have to represent table structure. This could be one of sever

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072931529 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072929729 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072927613 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386216107 Partition spec before update: ``` "fields" : [ { "name" : "timestamp1_day", "transform" : "day", "source-id" : 2, "field-id" : 1000

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072921664 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] dmgcodevil commented on issue #6587: Wrong class, java.lang.Long, for object: 19367

2023-01-17 Thread GitBox
dmgcodevil commented on issue #6587: URL: https://github.com/apache/iceberg/issues/6587#issuecomment-1386213039 @RussellSpitzer , I just realized that compaction (Flink, Spark) stopped working after I updated my partition spec: https://bbgithub.dev.bloomberg.com/gist/rpliashkou/40f63

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072905855 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072900019 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -116,7 +116,7 @@ private FileIO fileIO(Table table) { return table.io(); } - priva

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
haizhou-zhao commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1072885938 ## docs/configuration.md: ## @@ -175,8 +175,13 @@ The HMS table locking is a 2-step process: | iceberg.hive.lock-heartbeat-interval-ms | 24 (4 min) | The h

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
haizhou-zhao commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1072884107 ## hive-metastore/src/main/java/org/apache/iceberg/hive/NoLock.java: ## @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
haizhou-zhao commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1072875908 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,533 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
amogh-jahagirdar commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072871952 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
amogh-jahagirdar commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072871952 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default

[GitHub] [iceberg] RussellSpitzer commented on issue #6613: Multiple SparkSessions interact with Iceberg Table

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6613: URL: https://github.com/apache/iceberg/issues/6613#issuecomment-1386116704 Well read load by default goes through the DataSource (IcebergSource) and the spark.sql version is going through "SparkCatalog". These should use the same underlying CatalogMa

[GitHub] [iceberg] Blake-Guo commented on issue #6613: Multiple SparkSessions interact with Iceberg Table

2023-01-17 Thread GitBox
Blake-Guo commented on issue #6613: URL: https://github.com/apache/iceberg/issues/6613#issuecomment-1386108491 Thanks @RussellSpitzer . So depending on if the user accesses the Iceberg Table through `sparkSession.sql` or `sparkSession.read().*.load(*)`, Spark's CatalogManager might load

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
haizhou-zhao commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1072851109 ## hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java: ## @@ -0,0 +1,533 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6365: Core: Add position deletes metadata table

2023-01-17 Thread GitBox
aokolnychyi commented on code in PR #6365: URL: https://github.com/apache/iceberg/pull/6365#discussion_r1072781218 ## core/src/main/java/org/apache/iceberg/PositionDeletesTable.java: ## @@ -0,0 +1,396 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072839234 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java: ## @@ -361,4 +368,25 @@ static FileInfo toFileInfo(ContentFile file) {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072839234 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java: ## @@ -361,4 +368,25 @@ static FileInfo toFileInfo(ContentFile file) {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072837947 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java: ## @@ -361,4 +368,25 @@ static FileInfo toFileInfo(ContentFile file) {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072828083 ## core/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFilesActionResult.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [iceberg] RussellSpitzer commented on issue #6613: Multiple SparkSessions interact with Iceberg Table

2023-01-17 Thread GitBox
RussellSpitzer commented on issue #6613: URL: https://github.com/apache/iceberg/issues/6613#issuecomment-1386049515 This is probably almost certainly a Spark SQLConf weirdness. I've seen stranger behaviors here too depending on whether you are using a notebook, repl, or application code. If

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
amogh-jahagirdar commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1386049263 Raised a PR to @namrathamyske branch for replace partition and a bug fix for validation during overwrite https://github.com/namrathamyske/iceberg/pull/20/files -- This is an a

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-17 Thread GitBox
huaxingao commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1072805278 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.or

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072805060 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] Blake-Guo opened a new issue, #6613: Multiple SparkSessions interact with Iceberg Table

2023-01-17 Thread GitBox
Blake-Guo opened a new issue, #6613: URL: https://github.com/apache/iceberg/issues/6613 ### Apache Iceberg version 0.12.1 ### Query engine Spark ### Please describe the bug 🐞 I explored the multiple SparkSessions (to connect to different data sources/data c

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072798915 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072798915 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] pvary commented on pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
pvary commented on PR #6605: URL: https://github.com/apache/iceberg/pull/6605#issuecomment-1386019455 For the record #6570 has not been merged yet. I was not clear enough, sorry for that -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072678579 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -165,8 +165,13 @@ public void initialize(String name, Map unresolved) { Catalo

[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Parameterize token refresh retry attemps & add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072678579 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -165,8 +165,13 @@ public void initialize(String name, Map unresolved) { Catalo

[GitHub] [iceberg] vapography closed pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
vapography closed pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations URL: https://github.com/apache/iceberg/pull/6605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [iceberg] vapography commented on pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
vapography commented on PR #6605: URL: https://github.com/apache/iceberg/pull/6605#issuecomment-1385829209 Yes, I have completely missed scenario with multiple hive configurations. Great that issue is already resolved by #6570. Closing PR. -- This is an automated message from the Apa

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072512846 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6612: URL: https://github.com/apache/iceberg/pull/6612#discussion_r1072544944 ## format/view-spec.md: ## @@ -107,7 +107,7 @@ This type of representation stores the original view definition in SQL and its S | Required/Optional | Field Name |

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072532309 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default catal

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072532309 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default catal

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
amogh-jahagirdar opened a new pull request, #6612: URL: https://github.com/apache/iceberg/pull/6612 In the API/data model layer we call the field query now instead of "sql" https://github.com/apache/iceberg/pull/4925/files#diff-ec2ce8e1a850dc93199ebfe18a04747589a102882e052e9a83dd6602e366ac7a

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072512846 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072503036 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedGenericRecord.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6611: Spec: Make schema-id required for SQL View Representation

2023-01-17 Thread GitBox
amogh-jahagirdar opened a new pull request, #6611: URL: https://github.com/apache/iceberg/pull/6611 This change makes schema ID a required field for SQL View Representations. My rationale is that the Iceberg View specification should always have a well defined schema for a SQL View represen

[GitHub] [iceberg] nastra commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
nastra commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072480919 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -302,17 +317,59 @@ public static OAuthTokenResponse tokenResponseFromJson(JsonNode json) {

[GitHub] [iceberg] nastra commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
nastra commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072469640 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -376,5 +450,201 @@ public Pair refresh(RESTClient client) { return null; } + +

  1   2   >