[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-05 Thread via GitHub
stevenzwu commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096950110 ## aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java: ## @@ -0,0 +1,404 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] slfan1989 commented on pull request #6735: replace-parquet-deprecated-method

2023-02-05 Thread via GitHub
slfan1989 commented on PR #6735: URL: https://github.com/apache/iceberg/pull/6735#issuecomment-1418524098 @Fokko Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1096955753 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -109,6 +110,10 @@ public void initialize(String name, Map properties) {

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1096956705 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -45,6 +45,7 @@ public class SnowflakeCatalog extends BaseMetastoreCatalog

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1096958848 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -45,6 +45,7 @@ public class SnowflakeCatalog extends BaseMetastoreCatalog

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-05 Thread via GitHub
JonasJ-ap commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096958822 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vis

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1096959777 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -109,6 +110,10 @@ public void initialize(String name, Map properties) {

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1096964053 ## build.gradle: ## @@ -438,6 +442,77 @@ project(':iceberg-aws') { } } +project(':iceberg-delta-lake') { + // use integration test since we can take advanta

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-05 Thread via GitHub
JonasJ-ap commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096965051 ## aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java: ## @@ -0,0 +1,404 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-05 Thread via GitHub
JonasJ-ap commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096965051 ## aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java: ## @@ -0,0 +1,404 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1096967602 ## build.gradle: ## @@ -438,6 +442,77 @@ project(':iceberg-aws') { } } +project(':iceberg-delta-lake') { + // use integration test since we can take advanta

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1096968759 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6449: Delta: Support Snapshot Delta Lake Table to Iceberg Table

2023-02-05 Thread via GitHub
danielcweeks commented on code in PR #6449: URL: https://github.com/apache/iceberg/pull/6449#discussion_r1096968759 ## delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [iceberg] 372242283 commented on issue #3466: "Manifest is missing" ValidationException when there have Concurrent applications to rewrite manifests

2023-02-05 Thread via GitHub
372242283 commented on issue #3466: URL: https://github.com/apache/iceberg/issues/3466#issuecomment-1418562275 I have the same problem, which happens occasionally,big guys, help me analyze it Thank you: version:iceberg1.1.0,spark3.1.3 CALL hive_catalog.system.rewrite_manifests('xx.tabl

[GitHub] [iceberg] nastra commented on a diff in pull request #6410: Configurable metrics reporter by catalog properties

2023-02-05 Thread via GitHub
nastra commented on code in PR #6410: URL: https://github.com/apache/iceberg/pull/6410#discussion_r1097016602 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -464,6 +464,11 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.namesp

[GitHub] [iceberg] nastra commented on pull request #6696: Build: Bump Arrow from 10.0.1 to 11.0.0

2023-02-05 Thread via GitHub
nastra commented on PR #6696: URL: https://github.com/apache/iceberg/pull/6696#issuecomment-1418623938 @ajantha-bhat the results are stored and linked under [benchmark-results](https://github.com/ajantha-bhat/iceberg/suites/10704631127/artifacts/541607117) for each benchmark run. -- This

[GitHub] [iceberg] nastra commented on pull request #6687: Build: Bump jackson-annotations from 2.14.1 to 2.14.2

2023-02-05 Thread via GitHub
nastra commented on PR #6687: URL: https://github.com/apache/iceberg/pull/6687#issuecomment-1418628552 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [iceberg] nastra commented on issue #6751: iceberg roadmap

2023-02-05 Thread via GitHub
nastra commented on issue #6751: URL: https://github.com/apache/iceberg/issues/6751#issuecomment-1418634674 The Iceberg roadmap can be found under https://iceberg.apache.org/roadmap/. It contains a high-level overview of what the community is working on. -- This is an automated message fr

[GitHub] [iceberg] nastra closed issue #6751: iceberg roadmap

2023-02-05 Thread via GitHub
nastra closed issue #6751: iceberg roadmap URL: https://github.com/apache/iceberg/issues/6751 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-uns

[GitHub] [iceberg] chenjunjiedada opened a new pull request, #6753: Flink: use projection when delete records in upsert mode

2023-02-05 Thread via GitHub
chenjunjiedada opened a new pull request, #6753: URL: https://github.com/apache/iceberg/pull/6753 The equality delete writer is built with projected schema in upsert mode, but now it passes row data to it without projection and thus causes problems in case of the equality column is not the

[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

2023-02-05 Thread via GitHub
singhpk234 commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097020044 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala: ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache

[GitHub] [iceberg] ajantha-bhat commented on pull request #6696: Build: Bump Arrow from 10.0.1 to 11.0.0

2023-02-05 Thread via GitHub
ajantha-bhat commented on PR #6696: URL: https://github.com/apache/iceberg/pull/6696#issuecomment-1418643318 ``` Benchmark [Arrow-11.0.0] Mode Cnt Score Error Units SparkP

[GitHub] [iceberg] ajantha-bhat commented on pull request #6696: Build: Bump Arrow from 10.0.1 to 11.0.0

2023-02-05 Thread via GitHub
ajantha-bhat commented on PR #6696: URL: https://github.com/apache/iceberg/pull/6696#issuecomment-1418644100 https://user-images.githubusercontent.com/5889404/216913511-533f8206-c974-4e78-b2a7-4416c04a1c06.png";> -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [iceberg] gaborkaszab commented on pull request #6074: API,Core: SnapshotManager to be created through Transaction

2023-02-06 Thread via GitHub
gaborkaszab commented on PR #6074: URL: https://github.com/apache/iceberg/pull/6074#issuecomment-1418703342 Thanks @amogh-jahagirdar, @rdblue and @nastra for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [iceberg] tprelle opened a new issue, #6754: Be able to add extra snapshot metadata on sql statement with pyspark

2023-02-06 Thread via GitHub
tprelle opened a new issue, #6754: URL: https://github.com/apache/iceberg/issues/6754 ### Feature Request / Improvement Currently if you are using pyspark and sql statement you can not set extra snapshot metadata as it's only possible using a java lambda. ### Query engine

[GitHub] [iceberg] tprelle opened a new pull request, #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
tprelle opened a new pull request, #6755: URL: https://github.com/apache/iceberg/pull/6755 Currently if you are using pyspark and sql statement you can not set extra snapshot metadata as it's only possible using a java lambda. But we can use spark sql conf to retrieve metadata. It will

[GitHub] [iceberg] hililiwei commented on pull request #6637: Spark: Spark SQL Extensions for create tag

2023-02-06 Thread via GitHub
hililiwei commented on PR #6637: URL: https://github.com/apache/iceberg/pull/6637#issuecomment-1418726533 > hililiwei requested review from jackye1995 and removed request for flyrain and yyanyy Misclick. 😂 > Apart from the style changes for branch that can now be removed, I hav

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6637: Spark: Spark SQL Extensions for create tag

2023-02-06 Thread via GitHub
hililiwei commented on code in PR #6637: URL: https://github.com/apache/iceberg/pull/6637#discussion_r1097099267 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala: ## @@ -128,6 +133,36 @@ class Iceb

[GitHub] [iceberg] hililiwei commented on a diff in pull request #5029: Flink: Use Tag or Branch to scan data.

2023-02-06 Thread via GitHub
hililiwei commented on code in PR #5029: URL: https://github.com/apache/iceberg/pull/5029#discussion_r1097116000 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/StreamingMonitorFunction.java: ## @@ -124,11 +126,33 @@ public void initializeState(FunctionInitial

[GitHub] [iceberg] xuzhiwen1255 commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

2023-02-06 Thread via GitHub
xuzhiwen1255 commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1418765884 @pvary Can you take a look? Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] zoucao opened a new issue, #6756: The out-of-order problem occurs around the process of recovery

2023-02-06 Thread via GitHub
zoucao opened a new issue, #6756: URL: https://github.com/apache/iceberg/issues/6756 ### Apache Iceberg version 1.1.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 Recently, we face an out-of-order problem when consuming records from

[GitHub] [iceberg] kmozaid commented on a diff in pull request #6410: Configurable metrics reporter by catalog properties

2023-02-06 Thread via GitHub
kmozaid commented on code in PR #6410: URL: https://github.com/apache/iceberg/pull/6410#discussion_r1097194406 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -464,6 +464,11 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.names

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6661: Core: Support delete file stats in partitions metadata table

2023-02-06 Thread via GitHub
gaborkaszab commented on code in PR #6661: URL: https://github.com/apache/iceberg/pull/6661#discussion_r1097102496 ## core/src/main/java/org/apache/iceberg/PartitionsTable.java: ## @@ -47,7 +48,11 @@ public class PartitionsTable extends BaseMetadataTable { Types.Nes

[GitHub] [iceberg] gaborkaszab commented on issue #6257: Partitions metadata table shows old partitions

2023-02-06 Thread via GitHub
gaborkaszab commented on issue #6257: URL: https://github.com/apache/iceberg/issues/6257#issuecomment-1418896382 Thanks for the feedback, @szehon-ho! About documenting this phenomenon: I found the [spark-queries page](https://iceberg.apache.org/docs/latest/spark-queries/#partitions) that

[GitHub] [iceberg] bluzy commented on issue #6750: Failed to get table info from metastore using impersonation

2023-02-06 Thread via GitHub
bluzy commented on issue #6750: URL: https://github.com/apache/iceberg/issues/6750#issuecomment-1419010527 @szehon-ho RetryingMetaStoreClient added on https://github.com/apache/iceberg/pull/3099 But I doubt the retry logic doesn't change own ugi if another user handle it. I guess tur

[GitHub] [iceberg] RussellSpitzer commented on issue #3466: "Manifest is missing" ValidationException when there have Concurrent applications to rewrite manifests

2023-02-06 Thread via GitHub
RussellSpitzer commented on issue #3466: URL: https://github.com/apache/iceberg/issues/3466#issuecomment-1419098480 As I wrote above the issue os that the rewrite command becomes out of date while running so it fails. At least this is my hypothesis above -- This is an automated message fr

[GitHub] [iceberg] Hayder-Aziz-cardano commented on issue #6564: Python write support

2023-02-06 Thread via GitHub
Hayder-Aziz-cardano commented on issue #6564: URL: https://github.com/apache/iceberg/issues/6564#issuecomment-1419162378 it would be really nice to get it working with polars (which utilises pyarrow / connectorx) as it provides the possibility of running execution services without large mon

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

2023-02-06 Thread via GitHub
gaborkaszab commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1097453375 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pro

[GitHub] [iceberg] huaxingao commented on pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
huaxingao commented on PR #6622: URL: https://github.com/apache/iceberg/pull/6622#issuecomment-1419307821 @rdblue @aokolnychyi I have addressed all the comments. Could you please take a look again? Thanks a lot! -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [iceberg] RussellSpitzer opened a new issue, #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
RussellSpitzer opened a new issue, #6758: URL: https://github.com/apache/iceberg/issues/6758 ### Apache Iceberg version 1.1.0 (latest release) ### Query engine None ### Please describe the bug 🐞 An interesting thing we ran into: Our FileIo API contain

[GitHub] [iceberg] RussellSpitzer commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
RussellSpitzer commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419311188 @amogh-jahagirdar + @jackye1995 Do you have any thoughts on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [iceberg] jackye1995 commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
jackye1995 commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419396445 So internally we have a fix, where we extend Hadoop `Path` to work with double slash instead of the posix standard, and use that Path object for Iceberg for the specific `FileSyst

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097705742 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala: ## @@ -0,0 +1,50 @@ +/* + * Licensed to the A

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6742: support registerTable in GlueCatalog

2023-02-06 Thread via GitHub
jackye1995 commented on code in PR #6742: URL: https://github.com/apache/iceberg/pull/6742#discussion_r1097730419 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java: ## @@ -437,6 +439,44 @@ public void renameTable(TableIdentifier from, TableIdentifier to) {

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6621: [HiveCatalog] Support Altering and Dropping Table Ownership

2023-02-06 Thread via GitHub
haizhou-zhao commented on code in PR #6621: URL: https://github.com/apache/iceberg/pull/6621#discussion_r1097737870 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -494,6 +494,17 @@ private void setHmsTableParameters( // remove any pr

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
stevenzwu commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097776431 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vis

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6757: API,Core: Multi-Table transactions

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1097784829 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalogTransaction.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] RussellSpitzer commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
RussellSpitzer commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419591723 https://github.com/apache/iceberg/blob/cecb10bb8ab0458fb3f6a650692a8e432f08cbd2/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L226 I think it makes s

[GitHub] [iceberg] RussellSpitzer commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
RussellSpitzer commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419594072 For the Spec I was just considering something like "path" a path is a POSIX normalized path -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [iceberg] amogh-jahagirdar commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419597451 Mostly agree with @jackye1995 here. I think at minimum, we for sure should remove trailing slash when writing any file that's part of the Iceberg table. As for the sp

[GitHub] [iceberg] manisin commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-06 Thread via GitHub
manisin commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1097808842 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -109,6 +110,10 @@ public void initialize(String name, Map properties) {

[GitHub] [iceberg] manisin commented on a diff in pull request #6740: Add application identifier for Snowflake JDBC driver

2023-02-06 Thread via GitHub
manisin commented on code in PR #6740: URL: https://github.com/apache/iceberg/pull/6740#discussion_r1097809645 ## snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java: ## @@ -45,6 +45,7 @@ public class SnowflakeCatalog extends BaseMetastoreCatalog impl

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097864293 ## aws/src/main/java/org/apache/iceberg/aws/ApacheHttpClientConfigurations.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097868200 ## aws/src/main/java/org/apache/iceberg/aws/ApacheHttpClientConfigurations.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097866781 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -632,18 +628,11 @@ public class AwsProperties implements Serializable { */ public static

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097870631 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vi

[GitHub] [iceberg] rdblue commented on pull request #6757: API, Core: Multi-Table transactions

2023-02-06 Thread via GitHub
rdblue commented on PR #6757: URL: https://github.com/apache/iceberg/pull/6757#issuecomment-1419693712 @nastra, can you write up a design doc? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6757: API, Core: Multi-Table transactions

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6757: URL: https://github.com/apache/iceberg/pull/6757#discussion_r1097784829 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalogTransaction.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] rdblue merged pull request #6749: Prevent RESTCatalog AuthSession from expiring

2023-02-06 Thread via GitHub
rdblue merged PR #6749: URL: https://github.com/apache/iceberg/pull/6749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6749: Prevent RESTCatalog AuthSession from expiring

2023-02-06 Thread via GitHub
rdblue commented on PR #6749: URL: https://github.com/apache/iceberg/pull/6749#issuecomment-1419729043 Good catch, thanks for getting this fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [iceberg] rdblue commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1097905950 ## core/src/main/java/org/apache/iceberg/util/PropertyUtil.java: ## @@ -120,6 +120,27 @@ public static Map propertiesWithPrefix( .collect(Collectors.toMap(e ->

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097907479 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java: ## @@ -448,4 +448,37 @@ public void testExtraSnapshotMetadataWithSQL() thro

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097907981 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java: ## @@ -448,4 +448,37 @@ public void testExtraSnapshotMetadataWithSQL() thro

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097908288 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097908650 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097909160 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] rdblue commented on a diff in pull request #6755: [Spark] add extraSnapshotMetadata using sql conf

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6755: URL: https://github.com/apache/iceberg/pull/6755#discussion_r1097913008 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -192,6 +194,14 @@ public Map extraSnapshotMetadata() { } });

[GitHub] [iceberg] RussellSpitzer commented on issue #6758: S3FileIO Can Create Non-Posix Paths

2023-02-06 Thread via GitHub
RussellSpitzer commented on issue #6758: URL: https://github.com/apache/iceberg/issues/6758#issuecomment-1419751904 This is also an issue for data files here https://github.com/apache/iceberg/blob/223177faf955bd8f11864477da16013cf5d7cc75/core/src/main/java/org/apache/iceberg/LocationProvi

[GitHub] [iceberg] rdblue commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097916794 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [iceberg] rdblue commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097917630 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097920494 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097921708 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6752: Spark: DROP BRANCH SQL implementation

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6752: URL: https://github.com/apache/iceberg/pull/6752#discussion_r1097921035 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDropBranch.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Softw

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097921708 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097930735 ## api/src/main/java/org/apache/iceberg/expressions/BoundAggregate.java: ## @@ -44,4 +57,85 @@ public Type type() { return term().type(); } } + + public

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097936851 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushedFil

[GitHub] [iceberg] rdblue commented on a diff in pull request #6622: push down min/max/count to iceberg

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1097939971 ## api/src/main/java/org/apache/iceberg/expressions/CountNonNull.java: ## @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [iceberg] Fokko commented on issue #6708: Quick start docker-compose demo doesn't work

2023-02-06 Thread via GitHub
Fokko commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-1419813815 > py4j.protocol.Py4JNetworkError: Answer from Java side is empty Means that the Spark process has died. Do you see anything in the logs? -- This is an automated message from th

[GitHub] [iceberg] Fokko commented on issue #6708: Quick start docker-compose demo doesn't work

2023-02-06 Thread via GitHub
Fokko commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-1419814726 Can you run a `docker-compose pull` to make sure that you're running the latest version? We recently did some updates to the stack. -- This is an automated message from the Apache Gi

[GitHub] [iceberg] aokolnychyi commented on issue #6739: Integrate CRT with Iceberg S3 client

2023-02-06 Thread via GitHub
aokolnychyi commented on issue #6739: URL: https://github.com/apache/iceberg/issues/6739#issuecomment-1419851420 Sounds really promising! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [iceberg] Fokko commented on a diff in pull request #6646: Implement Support for DynamoDB Catalog

2023-02-06 Thread via GitHub
Fokko commented on code in PR #6646: URL: https://github.com/apache/iceberg/pull/6646#discussion_r1097181300 ## python/mkdocs/docs/configuration.md: ## @@ -85,3 +85,16 @@ catalog: default: type: glue ``` + +## DynamoDB Catalog + +If you want to use AWS DynamoDB as the c

[GitHub] [iceberg] Fokko commented on a diff in pull request #6646: Implement Support for DynamoDB Catalog

2023-02-06 Thread via GitHub
Fokko commented on code in PR #6646: URL: https://github.com/apache/iceberg/pull/6646#discussion_r1098000908 ## python/pyiceberg/catalog/dynamodb.py: ## @@ -0,0 +1,776 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [iceberg] Fokko commented on a diff in pull request #6646: Implement Support for DynamoDB Catalog

2023-02-06 Thread via GitHub
Fokko commented on code in PR #6646: URL: https://github.com/apache/iceberg/pull/6646#discussion_r1097196623 ## python/pyiceberg/catalog/__init__.py: ## @@ -431,3 +440,114 @@ def namespace_from(identifier: Union[str, Identifier]) -> Identifier: Identifier: Namespac

[GitHub] [iceberg] github-actions[bot] commented on issue #5418: Refactoring flink upsert ut test so the table is partitioned by date.

2023-02-06 Thread via GitHub
github-actions[bot] commented on issue #5418: URL: https://github.com/apache/iceberg/issues/5418#issuecomment-1419974962 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098049424 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098049871 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098050188 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098050503 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098050794 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098057989 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098058915 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098059479 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098059626 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098060389 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098061423 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

2023-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1098060983 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStri

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098062631 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-02-06 Thread via GitHub
rdblue commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1098063044 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

<    9   10   11   12   13   14   15   16   17   18   >