[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6578: Core: Allow retrieving response headers from last request

2023-01-16 Thread GitBox
danielcweeks commented on code in PR #6578: URL: https://github.com/apache/iceberg/pull/6578#discussion_r1071463846 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -64,6 +65,7 @@ public class HTTPClient implements RESTClient { private final CloseableHttpC

[GitHub] [iceberg] nastra commented on a diff in pull request #6578: Core: Allow retrieving response headers from last request

2023-01-16 Thread GitBox
nastra commented on code in PR #6578: URL: https://github.com/apache/iceberg/pull/6578#discussion_r1071479246 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -64,6 +65,7 @@ public class HTTPClient implements RESTClient { private final CloseableHttpClient

[GitHub] [iceberg] qinghui-xu commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail

2023-01-16 Thread GitBox
qinghui-xu commented on issue #6579: URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1384392111 Sure, I will be happy to contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] nastra commented on a diff in pull request #6578: Core: Allow retrieving response headers from last request

2023-01-16 Thread GitBox
nastra commented on code in PR #6578: URL: https://github.com/apache/iceberg/pull/6578#discussion_r1071479246 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -64,6 +65,7 @@ public class HTTPClient implements RESTClient { private final CloseableHttpClient

[GitHub] [iceberg] pvary merged pull request #6603: Flink: Backport: Improve unit tests for sink

2023-01-16 Thread GitBox
pvary merged PR #6603: URL: https://github.com/apache/iceberg/pull/6603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] pvary commented on pull request #6603: Flink: Backport: Improve unit tests for sink

2023-01-16 Thread GitBox
pvary commented on PR #6603: URL: https://github.com/apache/iceberg/pull/6603#issuecomment-1384450846 Merged to master. Thanks for the review @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [iceberg] deniskuzZ commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-16 Thread GitBox
deniskuzZ commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1071413371 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -809,168 +603,38 @@ private static boolean hiveEngineEnabled(TableMetadata me

[GitHub] [iceberg] namrathamyske commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-16 Thread GitBox
namrathamyske commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1384537219 @jackye1995 @amogh-jahagirdar Finished for the above producer implementations. I am yet to commit tests for BaseReplacePartitions. Let me know what you think! -- This is an autom

[GitHub] [iceberg] rdblue commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-16 Thread GitBox
rdblue commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071586466 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -335,6 +335,7 @@ public DeltaWriter createWriter(int partitionId,

[GitHub] [iceberg] rdblue commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-16 Thread GitBox
rdblue commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071586945 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -335,6 +335,7 @@ public DeltaWriter createWriter(int partitionId,

[GitHub] [iceberg] rdblue commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-16 Thread GitBox
rdblue commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071587465 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -335,10 +335,14 @@ public DeltaWriter createWriter(int partitionId

[GitHub] [iceberg] rdblue commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-16 Thread GitBox
rdblue commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071587694 ## core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java: ## @@ -143,12 +153,17 @@ public Builder format(FileFormat newFormat) { return this; } +

[GitHub] [iceberg] rdblue merged pull request #6357: Core: Remove TableOperations from metadata tables.

2023-01-16 Thread GitBox
rdblue merged PR #6357: URL: https://github.com/apache/iceberg/pull/6357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6357: Core: Remove TableOperations from metadata tables.

2023-01-16 Thread GitBox
rdblue commented on PR #6357: URL: https://github.com/apache/iceberg/pull/6357#issuecomment-1384578329 Thanks for the review, @szehon-ho! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [iceberg] rdblue commented on a diff in pull request #6357: Core: Remove TableOperations from metadata tables.

2023-01-16 Thread GitBox
rdblue commented on code in PR #6357: URL: https://github.com/apache/iceberg/pull/6357#discussion_r1071590953 ## core/src/main/java/org/apache/iceberg/BaseTable.java: ## @@ -229,17 +229,17 @@ public Transaction newTransaction() { @Override public FileIO io() { -retur

[GitHub] [iceberg] github-actions[bot] commented on issue #5314: Support precision of TimestampType

2023-01-16 Thread GitBox
github-actions[bot] commented on issue #5314: URL: https://github.com/apache/iceberg/issues/5314#issuecomment-1384687772 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] commented on issue #5243: iceberg + hive4.0

2023-01-16 Thread GitBox
github-actions[bot] commented on issue #5243: URL: https://github.com/apache/iceberg/issues/5243#issuecomment-1384687804 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] commented on issue #5231: Metadata file always gets created under /user/hive/ dir

2023-01-16 Thread GitBox
github-actions[bot] commented on issue #5231: URL: https://github.com/apache/iceberg/issues/5231#issuecomment-1384687822 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-16 Thread GitBox
amogh-jahagirdar commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1384706303 Great work @namrathamyske, just some minor comments but at least the core of it around the tests looks good to me. When you get a chance to update the ReplacePartition tests that

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-16 Thread GitBox
amogh-jahagirdar commented on code in PR #5234: URL: https://github.com/apache/iceberg/pull/5234#discussion_r1071651640 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -267,9 +267,10 @@ private Map summary(TableMetadata previous) { } Map previo

[GitHub] [iceberg] vapography opened a new pull request, #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-16 Thread GitBox
vapography opened a new pull request, #6605: URL: https://github.com/apache/iceberg/pull/6605 Fix thread leak in hive-metastore caused by new thread pool instance created for each HiveTableOperations instance -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [iceberg] rdblue commented on pull request #6489: Core: Improve token exchange handling when token expires

2023-01-16 Thread GitBox
rdblue commented on PR #6489: URL: https://github.com/apache/iceberg/pull/6489#issuecomment-1384717820 @nastra, I opened a PR against your branch with the remaining changes I think we should make. Please take a look: https://github.com/nastra/iceberg/pull/72 -- This is an automated messa

[GitHub] [iceberg] rdblue commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-16 Thread GitBox
rdblue commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1071659055 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -344,7 +382,20 @@ public Pair refresh(RESTClient client) { Tasks.foreach(ref)

[GitHub] [iceberg] cgpoh commented on issue #5256: S3 Unable to execute HTTP request: Timeout waiting for connection from pool

2023-01-16 Thread GitBox
cgpoh commented on issue #5256: URL: https://github.com/apache/iceberg/issues/5256#issuecomment-1384744063 @20100507 May I know how did you resolve this issue? I'm facing the same problem even when I set the s3.connection.maximum to 100. 1 Iceberg worker resolve this issue though. -- Th

[GitHub] [iceberg] yittg closed issue #4251: Proposal: Support partition with transform for Flink SQL

2023-01-16 Thread GitBox
yittg closed issue #4251: Proposal: Support partition with transform for Flink SQL URL: https://github.com/apache/iceberg/issues/4251 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-01-16 Thread GitBox
JonasJ-ap commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1071694629 ## delta-lake/src/integration/java/org/apache/iceberg/delta/SparkDeltaLakeSnapshotTestBase.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-16 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1071705366 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/ListDataIteratorBatcher.java: ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [iceberg] cgpoh opened a new issue, #6606: MinIO com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool

2023-01-16 Thread GitBox
cgpoh opened a new issue, #6606: URL: https://github.com/apache/iceberg/issues/6606 ### Apache Iceberg version 1.1.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 operating environment: Flink 1.15.2 Iceberg 1.1.0 Hadoop A

[GitHub] [iceberg] Fokko merged pull request #6593: Build: Bump moto from 4.0.13 to 4.1.0 in /python

2023-01-16 Thread GitBox
Fokko merged PR #6593: URL: https://github.com/apache/iceberg/pull/6593 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] pvary merged pull request #6602: Flink: Backport: Flink: Refactor sink tests to use HadoopCatalogResource

2023-01-16 Thread GitBox
pvary merged PR #6602: URL: https://github.com/apache/iceberg/pull/6602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] pvary commented on pull request #6602: Flink: Backport: Flink: Refactor sink tests to use HadoopCatalogResource

2023-01-16 Thread GitBox
pvary commented on PR #6602: URL: https://github.com/apache/iceberg/pull/6602#issuecomment-1384903152 Merged to master. Thanks for the review @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [iceberg] pvary commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-16 Thread GitBox
pvary commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1071794671 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/ListDataIteratorBatcher.java: ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [iceberg] pvary commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-16 Thread GitBox
pvary commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1071798337 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedGenericRecord.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Softwar

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

2023-01-16 Thread GitBox
youngxinler commented on code in PR #6554: URL: https://github.com/apache/iceberg/pull/6554#discussion_r1071834035 ## data/src/test/java/org/apache/iceberg/data/TestMetricsRowGroupFilter.java: ## @@ -341,6 +345,25 @@ public void testNotNaN() { Assert.assertTrue("Should read

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

2023-01-16 Thread GitBox
youngxinler commented on code in PR #6554: URL: https://github.com/apache/iceberg/pull/6554#discussion_r1071834035 ## data/src/test/java/org/apache/iceberg/data/TestMetricsRowGroupFilter.java: ## @@ -341,6 +345,25 @@ public void testNotNaN() { Assert.assertTrue("Should read

[GitHub] [iceberg] arminnajafi commented on pull request #6445: Python: Mock home and root folder when running `test_missing_uri`

2023-01-17 Thread GitBox
arminnajafi commented on PR #6445: URL: https://github.com/apache/iceberg/pull/6445#issuecomment-1384992306 I don't think this change is not cutting it. I think it because before we get to the patch line: ``` # mock to prevent parsing ~/.pyiceberg.yaml or {PYICEBERG_HOME}/.pyice

[GitHub] [iceberg] Fokko commented on a diff in pull request #6590: Python: Add sql command to the CLI

2023-01-17 Thread GitBox
Fokko commented on code in PR #6590: URL: https://github.com/apache/iceberg/pull/6590#discussion_r1071885001 ## python/pyiceberg/table/__init__.py: ## @@ -328,14 +331,23 @@ def plan_files(self) -> Iterator[FileScanTask]: partition_evaluators: Dict[int, Callable[[DataF

[GitHub] [iceberg] arminnajafi commented on pull request #6445: Python: Mock home and root folder when running `test_missing_uri`

2023-01-17 Thread GitBox
arminnajafi commented on PR #6445: URL: https://github.com/apache/iceberg/pull/6445#issuecomment-1385007961 One more mock is needed to make it work: ``` with mock.patch('pyiceberg.catalog._ENV_CONFIG', Config()): ``` ``` def test_missing_uri(empty_home_dir_path: str

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1071910972 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -303,6 +303,9 @@ private TableProperties() {} public static final String ENGINE_HIVE_ENABLED = "en

[GitHub] [iceberg] arminnajafi opened a new pull request, #6607: Python: Fix `test_missing_uri` unit test

2023-01-17 Thread GitBox
arminnajafi opened a new pull request, #6607: URL: https://github.com/apache/iceberg/pull/6607 When running `make test` on mainline branch `test_missing_uri` unit test fails with: ``` empty_home_dir_path = '/private/var/folders/wz/yzqdwvrx0cj1j1h5c3g59rm4gr/T/pytest-of-najarmi

[GitHub] [iceberg] arminnajafi commented on pull request #6445: Python: Mock home and root folder when running `test_missing_uri`

2023-01-17 Thread GitBox
arminnajafi commented on PR #6445: URL: https://github.com/apache/iceberg/pull/6445#issuecomment-1385032472 https://github.com/apache/iceberg/pull/6607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [iceberg] arminnajafi commented on issue #6541: Python: Support DynamoDB catalog

2023-01-17 Thread GitBox
arminnajafi commented on issue #6541: URL: https://github.com/apache/iceberg/issues/6541#issuecomment-1385033795 - Read https://py.iceberg.apache.org/ cover to cover - Set up the repo in IntelliJ with a Poetry env -- This is an automated message from the Apache Git Servi

[GitHub] [iceberg] arminnajafi commented on issue #6541: Python: Support DynamoDB catalog

2023-01-17 Thread GitBox
arminnajafi commented on issue #6541: URL: https://github.com/apache/iceberg/issues/6541#issuecomment-1385035151 However I realized when running `make test` on mainline branch `test_missing_uri` unit test fails. Here is the PR for the unit test fix: https://github.com/apache/icebe

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1071920474 ## docs/configuration.md: ## @@ -175,8 +175,13 @@ The HMS table locking is a 2-step process: | iceberg.hive.lock-heartbeat-interval-ms | 24 (4 min) | The heartbea

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1071920742 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -809,168 +603,38 @@ private static boolean hiveEngineEnabled(TableMetadata metada

[GitHub] [iceberg] YesOrNo828 opened a new pull request, #6608: Docs: Update the description of binary and varbinary type in the type conversion section

2023-01-17 Thread GitBox
YesOrNo828 opened a new pull request, #6608: URL: https://github.com/apache/iceberg/pull/6608 Update the description of binary and varbinary types when users convert flink types to iceberg types in the [Type Conversion](https://iceberg.apache.org/docs/latest/flink/#flink-to-iceberg) sectio

[GitHub] [iceberg] nastra commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
nastra commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071924681 ## core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java: ## @@ -143,12 +153,17 @@ public Builder format(FileFormat newFormat) { return this; } +

[GitHub] [iceberg] nastra commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
nastra commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071925326 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -335,6 +335,7 @@ public DeltaWriter createWriter(int partitionId,

[GitHub] [iceberg] nastra commented on a diff in pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
nastra commented on code in PR #6569: URL: https://github.com/apache/iceberg/pull/6569#discussion_r1071926775 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -335,10 +335,14 @@ public DeltaWriter createWriter(int partitionId

[GitHub] [iceberg] nastra commented on pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
nastra commented on PR #6569: URL: https://github.com/apache/iceberg/pull/6569#issuecomment-1385046987 @rdblue thanks for the review. I've removed all `.suffix("data")` calls so that we only have `.suffix("deletes")` now. -- This is an automated message from the Apache Git Service. To res

[GitHub] [iceberg] pvary commented on a diff in pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on code in PR #6570: URL: https://github.com/apache/iceberg/pull/6570#discussion_r1071927823 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -809,168 +603,38 @@ private static boolean hiveEngineEnabled(TableMetadata metada

[GitHub] [iceberg] pvary commented on pull request #6570: Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882

2023-01-17 Thread GitBox
pvary commented on PR #6570: URL: https://github.com/apache/iceberg/pull/6570#issuecomment-1385051268 > LGTM, thanks for the path Peter, that should definitely improve the Hive iceberg performance. Let me know what kind of help is needed from the Hive folks Hi @deniskuzZ, good to hear

[GitHub] [iceberg] YesOrNo828 commented on pull request #6608: Docs: Update the description of binary and varbinary type in the type conversion section

2023-01-17 Thread GitBox
YesOrNo828 commented on PR #6608: URL: https://github.com/apache/iceberg/pull/6608#issuecomment-1385052348 @stevenzwu @hililiwei Do you have time to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] pvary commented on pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
pvary commented on PR #6605: URL: https://github.com/apache/iceberg/pull/6605#issuecomment-1385069134 Funny thing that I have found the same issue when running the spark tests against my PR: #6570. #6570 would fix this too by recreating the `hiveLockHeartbeatExecutorService` for every Lo

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

2023-01-17 Thread GitBox
youngxinler commented on code in PR #6554: URL: https://github.com/apache/iceberg/pull/6554#discussion_r1071834035 ## data/src/test/java/org/apache/iceberg/data/TestMetricsRowGroupFilter.java: ## @@ -341,6 +345,25 @@ public void testNotNaN() { Assert.assertTrue("Should read

[GitHub] [iceberg] nastra commented on pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
nastra commented on PR #6489: URL: https://github.com/apache/iceberg/pull/6489#issuecomment-1385226063 > @nastra, I opened a PR against your branch with the remaining changes I think we should make. Please take a look: [nastra#72](https://github.com/nastra/iceberg/pull/72) Thanks @rd

[GitHub] [iceberg] ggershinsky commented on a diff in pull request #6485: API: New KMS Client Interface

2023-01-17 Thread GitBox
ggershinsky commented on code in PR #6485: URL: https://github.com/apache/iceberg/pull/6485#discussion_r1072105579 ## api/src/main/java/org/apache/iceberg/encryption/KeyManagementClient.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] nastra closed issue #6453: Iceberg delete-append causing snapshot error

2023-01-17 Thread GitBox
nastra closed issue #6453: Iceberg delete-append causing snapshot error URL: https://github.com/apache/iceberg/issues/6453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [iceberg] nastra commented on issue #6453: Iceberg delete-append causing snapshot error

2023-01-17 Thread GitBox
nastra commented on issue #6453: URL: https://github.com/apache/iceberg/issues/6453#issuecomment-1385303484 Closing this for now. Please feel free to re-open if necessary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg] ggershinsky commented on a diff in pull request #2639: Parquet: Support parquet modular encryption

2023-01-17 Thread GitBox
ggershinsky commented on code in PR #2639: URL: https://github.com/apache/iceberg/pull/2639#discussion_r1072111445 ## core/src/test/java/org/apache/iceberg/encryption/EncryptionTestUtils.java: ## @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

[GitHub] [iceberg] ggershinsky commented on a diff in pull request #5544: Encryption integration and test

2023-01-17 Thread GitBox
ggershinsky commented on code in PR #5544: URL: https://github.com/apache/iceberg/pull/5544#discussion_r1072113069 ## core/src/main/java/org/apache/iceberg/encryption/DefaultEncryptionManagerFactory.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] ggershinsky commented on a diff in pull request #5544: Encryption integration and test

2023-01-17 Thread GitBox
ggershinsky commented on code in PR #5544: URL: https://github.com/apache/iceberg/pull/5544#discussion_r1072119326 ## core/src/main/java/org/apache/iceberg/encryption/DefaultEncryptionManagerFactory.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] [iceberg] hililiwei opened a new pull request, #6610: Flink: Port Support inspecting metadata table to Flink 1.14 & 1.15

2023-01-17 Thread GitBox
hililiwei opened a new pull request, #6610: URL: https://github.com/apache/iceberg/pull/6610 ``` ➜ iceberg git:(inspecting2) ✗ git diff --no-index flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/ flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/ import org.apache.f

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
hililiwei commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072136039 ## docs/flink-getting-started.md: ## @@ -613,6 +613,47 @@ env.execute("Test Iceberg Streaming Read"); There are other options that we could set by Java API, please s

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
hililiwei commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072142396 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedGenericRecord.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
hililiwei commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072149460 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] danielcweeks merged pull request #6589: Core: Fix BaseMetastoreCatalog#registerTable to return full table name

2023-01-17 Thread GitBox
danielcweeks merged PR #6589: URL: https://github.com/apache/iceberg/pull/6589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[GitHub] [iceberg] findepi commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-17 Thread GitBox
findepi commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1072412666 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.org/

[GitHub] [iceberg] nastra commented on a diff in pull request #6578: Core: Allow retrieving response headers from last request

2023-01-17 Thread GitBox
nastra commented on code in PR #6578: URL: https://github.com/apache/iceberg/pull/6578#discussion_r1072436002 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -64,6 +65,7 @@ public class HTTPClient implements RESTClient { private final CloseableHttpClient

[GitHub] [iceberg] rdblue commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
rdblue commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072448584 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -302,17 +317,59 @@ public static OAuthTokenResponse tokenResponseFromJson(JsonNode json) {

[GitHub] [iceberg] rdblue commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
rdblue commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072451113 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -376,5 +450,201 @@ public Pair refresh(RESTClient client) { return null; } + +

[GitHub] [iceberg] rdblue merged pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
rdblue merged PR #6489: URL: https://github.com/apache/iceberg/pull/6489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
rdblue commented on PR #6489: URL: https://github.com/apache/iceberg/pull/6489#issuecomment-1385717405 I think that we need to fix https://github.com/apache/iceberg/pull/6489/files#r1072451113 in a follow-up, but that's minor so I'm going to merge this to unblock the next steps. Thanks, @n

[GitHub] [iceberg] rdblue commented on pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
rdblue commented on PR #6569: URL: https://github.com/apache/iceberg/pull/6569#issuecomment-1385719707 Thanks for picking this up, @nastra! Looks great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] rdblue merged pull request #6569: Spark: Add the query ID to file names

2023-01-17 Thread GitBox
rdblue merged PR #6569: URL: https://github.com/apache/iceberg/pull/6569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] jzhuge commented on pull request #6565: Core: View history entry core implementation

2023-01-17 Thread GitBox
jzhuge commented on PR #6565: URL: https://github.com/apache/iceberg/pull/6565#issuecomment-1385720776 LGTM! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [iceberg] stevenzwu merged pull request #6610: Flink: Port Support inspecting metadata table to Flink 1.14 & 1.15

2023-01-17 Thread GitBox
stevenzwu merged PR #6610: URL: https://github.com/apache/iceberg/pull/6610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg] stevenzwu commented on pull request #6610: Flink: Port Support inspecting metadata table to Flink 1.14 & 1.15

2023-01-17 Thread GitBox
stevenzwu commented on PR #6610: URL: https://github.com/apache/iceberg/pull/6610#issuecomment-1385727800 thanks @hililiwei for the backport of PR #6222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] rdblue commented on a diff in pull request #6609: Core: Parameterize token refresh retry attemps & add test for token expiration during refresh

2023-01-17 Thread GitBox
rdblue commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072467805 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -165,8 +165,13 @@ public void initialize(String name, Map unresolved) { Catalo

[GitHub] [iceberg] nastra commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
nastra commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072469640 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -376,5 +450,201 @@ public Pair refresh(RESTClient client) { return null; } + +

[GitHub] [iceberg] nastra commented on a diff in pull request #6489: Core: Improve token exchange handling when token expires

2023-01-17 Thread GitBox
nastra commented on code in PR #6489: URL: https://github.com/apache/iceberg/pull/6489#discussion_r1072480919 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -302,17 +317,59 @@ public static OAuthTokenResponse tokenResponseFromJson(JsonNode json) {

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6611: Spec: Make schema-id required for SQL View Representation

2023-01-17 Thread GitBox
amogh-jahagirdar opened a new pull request, #6611: URL: https://github.com/apache/iceberg/pull/6611 This change makes schema ID a required field for SQL View Representations. My rationale is that the Iceberg View specification should always have a well defined schema for a SQL View represen

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072503036 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedGenericRecord.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072512846 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
amogh-jahagirdar opened a new pull request, #6612: URL: https://github.com/apache/iceberg/pull/6612 In the API/data model layer we call the field query now instead of "sql" https://github.com/apache/iceberg/pull/4925/files#diff-ec2ce8e1a850dc93199ebfe18a04747589a102882e052e9a83dd6602e366ac7a

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072532309 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default catal

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1072532309 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -36,17 +38,21 @@ default Type type() { String dialect(); /** The default catal

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6612: Spec: View spec rename sql field to query

2023-01-17 Thread GitBox
jackye1995 commented on code in PR #6612: URL: https://github.com/apache/iceberg/pull/6612#discussion_r1072544944 ## format/view-spec.md: ## @@ -107,7 +107,7 @@ This type of representation stores the original view definition in SQL and its S | Required/Optional | Field Name |

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6584: Flink: support reading as Avro GenericRecord for FLIP-27 IcebergSource

2023-01-17 Thread GitBox
stevenzwu commented on code in PR #6584: URL: https://github.com/apache/iceberg/pull/6584#discussion_r1072512846 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/reader/AvroGenericRecordReaderFunction.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Sof

[GitHub] [iceberg] vapography commented on pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
vapography commented on PR #6605: URL: https://github.com/apache/iceberg/pull/6605#issuecomment-1385829209 Yes, I have completely missed scenario with multiple hive configurations. Great that issue is already resolved by #6570. Closing PR. -- This is an automated message from the Apa

[GitHub] [iceberg] vapography closed pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
vapography closed pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations URL: https://github.com/apache/iceberg/pull/6605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Parameterize token refresh retry attemps & add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072678579 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -165,8 +165,13 @@ public void initialize(String name, Map unresolved) { Catalo

[GitHub] [iceberg] nastra commented on a diff in pull request #6609: Core: Add test for token expiration during refresh

2023-01-17 Thread GitBox
nastra commented on code in PR #6609: URL: https://github.com/apache/iceberg/pull/6609#discussion_r1072678579 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -165,8 +165,13 @@ public void initialize(String name, Map unresolved) { Catalo

[GitHub] [iceberg] pvary commented on pull request #6605: Fix thread leak in hive-metastore caused by thread pool per HiveTableOperations

2023-01-17 Thread GitBox
pvary commented on PR #6605: URL: https://github.com/apache/iceberg/pull/6605#issuecomment-1386019455 For the record #6570 has not been merged yet. I was not clear enough, sorry for that -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072798915 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072798915 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] Blake-Guo opened a new issue, #6613: Multiple SparkSessions interact with Iceberg Table

2023-01-17 Thread GitBox
Blake-Guo opened a new issue, #6613: URL: https://github.com/apache/iceberg/issues/6613 ### Apache Iceberg version 0.12.1 ### Query engine Spark ### Please describe the bug 🐞 I explored the multiple SparkSessions (to connect to different data sources/data c

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

2023-01-17 Thread GitBox
RussellSpitzer commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1072805060 ## api/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFiles.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-17 Thread GitBox
huaxingao commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1072805278 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.or

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #5234: Core, API: BaseRowDelta, BaseOverwrite,BaseReplacePartitions, BaseRewrite to branch Impl

2023-01-17 Thread GitBox
amogh-jahagirdar commented on PR #5234: URL: https://github.com/apache/iceberg/pull/5234#issuecomment-1386049263 Raised a PR to @namrathamyske branch for replace partition and a bug fix for validation during overwrite https://github.com/namrathamyske/iceberg/pull/20/files -- This is an a

<    1   2   3   4   5   6   7   8   9   10   >