[GitHub] [iceberg] rdblue commented on a diff in pull request #6058: Core,Spark: Add metadata to Scan Report

2022-11-09 Thread GitBox
rdblue commented on code in PR #6058: URL: https://github.com/apache/iceberg/pull/6058#discussion_r1018469932 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -532,6 +537,24 @@ public final void initialize(String name, CaseInsensitiveStringMap

[GitHub] [iceberg] rdblue commented on a diff in pull request #6058: Core,Spark: Add metadata to Scan Report

2022-11-09 Thread GitBox
rdblue commented on code in PR #6058: URL: https://github.com/apache/iceberg/pull/6058#discussion_r1018470425 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -532,6 +537,24 @@ public final void initialize(String name, CaseInsensitiveStringMap

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018413283 ## api/src/main/java/org/apache/iceberg/FileScanTask.java: ## @@ -48,4 +48,9 @@ default boolean isFileScanTask() { default FileScanTask asFileScanTask() { r

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018475572 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018476321 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018477109 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on pull request #6012: Spark 3.3: Add a procedure to generate table changes

2022-11-09 Thread GitBox
aokolnychyi commented on PR #6012: URL: https://github.com/apache/iceberg/pull/6012#issuecomment-1309475241 Let me take a look today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018479165 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018485333 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018485570 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018485570 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018486198 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018487624 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018487624 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018488935 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -298,4 +324,33 @@ private static boolean compatibleTransforms(Transform t1, Transform

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018491119 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018492377 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018493002 ## api/src/main/java/org/apache/iceberg/FileScanTask.java: ## @@ -48,4 +48,9 @@ default boolean isFileScanTask() { default FileScanTask asFileScanTask() { retur

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018496077 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -298,4 +324,33 @@ private static boolean compatibleTransforms(Transform t1, Transform

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6163: Core: Method for building common partition type

2022-11-09 Thread GitBox
RussellSpitzer commented on code in PR #6163: URL: https://github.com/apache/iceberg/pull/6163#discussion_r1018499080 ## core/src/main/java/org/apache/iceberg/Partitioning.java: ## @@ -195,41 +198,68 @@ public Void alwaysNull(int fieldId, String sourceName, int sourceId) { }

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018502255 ## api/src/main/java/org/apache/iceberg/util/StructProjection.java: ## @@ -90,6 +90,13 @@ public static StructProjection createAllowMissing( private final StructProj

[GitHub] [iceberg] singhpk234 commented on a diff in pull request #5860: Spark: Fix QueryFailure when running RewriteManifestProcedure on Date partitioned table

2022-11-09 Thread GitBox
singhpk234 commented on code in PR #5860: URL: https://github.com/apache/iceberg/pull/5860#discussion_r1018512446 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteManifestsProcedure.java: ## @@ -75,6 +96,79 @@ public void testRewriteLar

[GitHub] [iceberg] singhpk234 commented on pull request #5860: Spark: Fix QueryFailure when running RewriteManifestProcedure on Date partitioned table

2022-11-09 Thread GitBox
singhpk234 commented on PR #5860: URL: https://github.com/apache/iceberg/pull/5860#issuecomment-1309542196 > I don't like running 4 tests in a loop inside one method when 2 test methods would do fine. Can you please remove the complexity of parameterizing this test? ACK, removed the

[GitHub] [iceberg] github-actions[bot] commented on issue #4723: [Feature Request][Flink][cdc]Iceberg FlinkSink support for writes multiple tables in a database, base on JoinedRowData which have value

2022-11-09 Thread GitBox
github-actions[bot] commented on issue #4723: URL: https://github.com/apache/iceberg/issues/4723#issuecomment-1309571874 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] commented on issue #4626: Get null values for for the nested field partition column

2022-11-09 Thread GitBox
github-actions[bot] commented on issue #4626: URL: https://github.com/apache/iceberg/issues/4626#issuecomment-1309571902 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] closed issue #3731: Cannot use RewriteDataFiles on Hadoop Tables

2022-11-09 Thread GitBox
github-actions[bot] closed issue #3731: Cannot use RewriteDataFiles on Hadoop Tables URL: https://github.com/apache/iceberg/issues/3731 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [iceberg] github-actions[bot] commented on issue #3731: Cannot use RewriteDataFiles on Hadoop Tables

2022-11-09 Thread GitBox
github-actions[bot] commented on issue #3731: URL: https://github.com/apache/iceberg/issues/3731#issuecomment-1309571969 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] daksha121 opened a new issue, #6165: Support for adding optional column to table identifiers

2022-11-09 Thread GitBox
daksha121 opened a new issue, #6165: URL: https://github.com/apache/iceberg/issues/6165 ### Feature Request / Improvement We have a scenario in which we need to add a new field to the table and add it to the identifiers as well (Currently the table has only one required column as the

[GitHub] [iceberg] rdblue commented on pull request #6161: Python: Load the S3 Session token

2022-11-09 Thread GitBox
rdblue commented on PR #6161: URL: https://github.com/apache/iceberg/pull/6161#issuecomment-1309603910 Should we change the S3 FileIO order to use ArrowFileIO by default as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] rdblue merged pull request #6161: Python: Load the S3 Session token

2022-11-09 Thread GitBox
rdblue merged PR #6161: URL: https://github.com/apache/iceberg/pull/6161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on a diff in pull request #6161: Python: Load the S3 Session token

2022-11-09 Thread GitBox
rdblue commented on code in PR #6161: URL: https://github.com/apache/iceberg/pull/6161#discussion_r1018547445 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +110,16 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntr

[GitHub] [iceberg] kekwan commented on issue #5970: Spark: Iceberg: java.io.InvalidClassException: org.apache.iceberg.Schema; local class incompatible: stream classdesc serialVersionUID = 332036701241

2022-11-09 Thread GitBox
kekwan commented on issue #5970: URL: https://github.com/apache/iceberg/issues/5970#issuecomment-1309628626 Hitting similar issue with `java.io.InvalidClassException: org.apache.iceberg.BaseFile; local class incompatible: stream classdesc serialVersionUID = 2686776604825259963, local class

[GitHub] [iceberg] lvyanquan opened a new pull request, #6166: Flink: Port #6049 to Flink 1.14 to add Sink options of compression properties

2022-11-09 Thread GitBox
lvyanquan opened a new pull request, #6166: URL: https://github.com/apache/iceberg/pull/6166 Flink: Port #6049 to Flink 1.14 in order to add Sink options of compression properties. This part of the code has been included in Flink 1.16 since #6092. -- This is an automated message fr

[GitHub] [iceberg] eyeryone commented on issue #6067: exec insert into (hive on spark),no erro log,but table no data

2022-11-09 Thread GitBox
eyeryone commented on issue #6067: URL: https://github.com/apache/iceberg/issues/6067#issuecomment-1309667485 > Hive on Spark is not supported/tested Also Hive 2.1.1 is also not supported/tested Could you use newer CDH/CDP version? ok ,thinks. If the version is not updated

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018632924 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -121,10 +128,66 @@ public static CloseableIterable> planTaskG Function weightFunc =

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018633047 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -121,10 +128,66 @@ public static CloseableIterable> planTaskG Function weightFunc =

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-09 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1018633696 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -121,10 +128,66 @@ public static CloseableIterable> planTaskG Function weightFunc =

[GitHub] [iceberg] SHuixo commented on issue #6104: Rewrite iceberg small files with flink succeeds but no snapshot is generated (V2 - upsert model)

2022-11-09 Thread GitBox
SHuixo commented on issue #6104: URL: https://github.com/apache/iceberg/issues/6104#issuecomment-1309843130 Think you @luoyuxia for your replay. Later, I tried again a few times and found that when the cumulative storage of iceberg data small files was relatively small, the flink 1.13

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6012: Spark 3.3: Add a procedure to generate table changes

2022-11-09 Thread GitBox
aokolnychyi commented on code in PR #6012: URL: https://github.com/apache/iceberg/pull/6012#discussion_r1018710056 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/GenerateChangesProcedure.java: ## @@ -0,0 +1,271 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-09 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018753250 ## .palantir/revapi.yml: ## @@ -1,4 +1,85 @@ acceptedBreaks: + "1.0.0": +org.apache.iceberg:iceberg-core: +- code: "java.class.defaultSerializationChanged" +

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-09 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018757312 ## .palantir/revapi.yml: ## @@ -11,15 +92,21 @@ acceptedBreaks: - code: "java.method.addedToInterface" new: "method java.lang.String org.apache.iceberg.expr

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-10 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018757312 ## .palantir/revapi.yml: ## @@ -11,15 +92,21 @@ acceptedBreaks: - code: "java.method.addedToInterface" new: "method java.lang.String org.apache.iceberg.expr

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-10 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018760620 ## .palantir/revapi.yml: ## @@ -11,15 +92,21 @@ acceptedBreaks: - code: "java.method.addedToInterface" new: "method java.lang.String org.apache.iceberg.expr

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-10 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018763350 ## .palantir/revapi.yml: ## @@ -1,4 +1,85 @@ acceptedBreaks: + "1.0.0": +org.apache.iceberg:iceberg-core: +- code: "java.class.defaultSerializationChanged" +

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-10 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018784561 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -269,6 +269,20 @@ public T post( return execute(Method.POST, path, null, body, responseType, h

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules & fix API breaks

2022-11-10 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1018802581 ## .palantir/revapi.yml: ## @@ -1,4 +1,85 @@ acceptedBreaks: + "1.0.0": +org.apache.iceberg:iceberg-core: +- code: "java.class.defaultSerializationChanged" +

[GitHub] [iceberg] pvary merged pull request #6166: Flink: Port #6049 to Flink 1.14 to add Sink options of compression properties

2022-11-10 Thread GitBox
pvary merged PR #6166: URL: https://github.com/apache/iceberg/pull/6166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] pvary commented on pull request #6166: Flink: Port #6049 to Flink 1.14 to add Sink options of compression properties

2022-11-10 Thread GitBox
pvary commented on PR #6166: URL: https://github.com/apache/iceberg/pull/6166#issuecomment-1309985875 Thanks @lvyanquan for taking care of this. Totally forgot to backport 😒 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] pvary commented on issue #6067: exec insert into (hive on spark),no erro log,but table no data

2022-11-10 Thread GitBox
pvary commented on issue #6067: URL: https://github.com/apache/iceberg/issues/6067#issuecomment-1309989646 I did not see progress on HoS side, and I think the Hive community decided to remove Spark as an execution engine, so I think there is not too much hope there. OTOH I know that

[GitHub] [iceberg] nastra commented on a diff in pull request #6058: Core,Spark: Add metadata to Scan Report

2022-11-10 Thread GitBox
nastra commented on code in PR #6058: URL: https://github.com/apache/iceberg/pull/6058#discussion_r1018846546 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -532,6 +537,24 @@ public final void initialize(String name, CaseInsensitiveStringMap

[GitHub] [iceberg] eyeryone commented on issue #6067: exec insert into (hive on spark),no erro log,but table no data

2022-11-10 Thread GitBox
eyeryone commented on issue #6067: URL: https://github.com/apache/iceberg/issues/6067#issuecomment-1310027231 > I did not see progress on HoS side, and I think the Hive community decided to remove Spark as an execution engine, so I think there is not too much hope there. > > OTOH I k

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-10 Thread GitBox
gaborkaszab commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1018774071 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -365,6 +374,13 @@ public boolean dropNamespace(Namespace namespace) { @Override

[GitHub] [iceberg] nastra commented on pull request #6133: [1.0] Dell: Fix client serialization bug.

2022-11-10 Thread GitBox
nastra commented on PR #6133: URL: https://github.com/apache/iceberg/pull/6133#issuecomment-1310103577 Going forward we'll be tagging & releasing from `master` but I don't think we'd want to delete `1.0.x`. That being said I think we can close this PR here since https://github.com/apache/ic

[GitHub] [iceberg] nastra closed pull request #6133: [1.0] Dell: Fix client serialization bug.

2022-11-10 Thread GitBox
nastra closed pull request #6133: [1.0] Dell: Fix client serialization bug. URL: https://github.com/apache/iceberg/pull/6133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [iceberg] Fokko commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
Fokko commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1018963278 ## python/pyiceberg/expressions/literals.py: ## @@ -58,25 +60,25 @@ timestamp_to_micros, timestamptz_to_micros, ) -from pyiceberg.utils.singleton import Single

[GitHub] [iceberg] Fokko commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
Fokko commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1017702146 ## python/pyiceberg/expressions/literals.py: ## @@ -125,81 +127,71 @@ def literal(value) -> Literal: @literal.register(bool) -def _(value: bool) -> Literal[bool]: +d

[GitHub] [iceberg] nastra commented on a diff in pull request #6167: Build: Add -DallVersions property that exposes all component versions

2022-11-10 Thread GitBox
nastra commented on code in PR #6167: URL: https://github.com/apache/iceberg/pull/6167#discussion_r1018975215 ## settings.gradle: ## @@ -52,6 +52,12 @@ project(':nessie').name = 'iceberg-nessie' project(':gcp').name = 'iceberg-gcp' project(':dell').name = 'iceberg-dell' +if

[GitHub] [iceberg] nastra commented on pull request #6102: Build: Fix build version

2022-11-10 Thread GitBox
nastra commented on PR #6102: URL: https://github.com/apache/iceberg/pull/6102#issuecomment-1310144291 @ajantha-bhat as we disussed yesterday in the Iceberg sync: Going forward we'll be tagging & releasing commits reachable from `master`, so I think this PR can be closed -- This is an au

[GitHub] [iceberg] Fokko merged pull request #6129: Build: Bump jackson-annotations from 2.13.4 to 2.14.0

2022-11-10 Thread GitBox
Fokko merged PR #6129: URL: https://github.com/apache/iceberg/pull/6129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6129: Build: Bump jackson-annotations from 2.13.4 to 2.14.0

2022-11-10 Thread GitBox
Fokko commented on PR #6129: URL: https://github.com/apache/iceberg/pull/6129#issuecomment-1310152192 Thanks everyone for the review πŸ‘πŸ» -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [iceberg] Fokko commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
Fokko commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1018963278 ## python/pyiceberg/expressions/literals.py: ## @@ -58,25 +60,25 @@ timestamp_to_micros, timestamptz_to_micros, ) -from pyiceberg.utils.singleton import Single

[GitHub] [iceberg] Fokko commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
Fokko commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1018963278 ## python/pyiceberg/expressions/literals.py: ## @@ -58,25 +60,25 @@ timestamp_to_micros, timestamptz_to_micros, ) -from pyiceberg.utils.singleton import Single

[GitHub] [iceberg] ggershinsky commented on pull request #5432: AES GCM Stream Spec

2022-11-10 Thread GitBox
ggershinsky commented on PR #5432: URL: https://github.com/apache/iceberg/pull/5432#issuecomment-1310159553 @rdblue I've sent a commit that addresses the review comments. Regarding the magic string - it's 4-byte long, so I've kept "AGS1" (only) there; but if it's a problem, I can replace

[GitHub] [iceberg] nastra closed pull request #6167: Build: Add -DallVersions property that exposes all component versions

2022-11-10 Thread GitBox
nastra closed pull request #6167: Build: Add -DallVersions property that exposes all component versions URL: https://github.com/apache/iceberg/pull/6167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6074: API,Core: SnapshotManager to be created through Transaction

2022-11-10 Thread GitBox
gaborkaszab commented on code in PR #6074: URL: https://github.com/apache/iceberg/pull/6074#discussion_r1019008623 ## core/src/main/java/org/apache/iceberg/SnapshotManager.java: ## @@ -22,6 +22,7 @@ public class SnapshotManager implements ManageSnapshots { + private final

[GitHub] [iceberg] nastra commented on pull request #6168: Build: Bump jackson from 2.11.4 to 2.14.0

2022-11-10 Thread GitBox
nastra commented on PR #6168: URL: https://github.com/apache/iceberg/pull/6168#issuecomment-1310173703 This currently fails with ``` Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.14.0 requires Jackson Databind version >= 2.14.0 and < 2.15.0

[GitHub] [iceberg] ajantha-bhat closed pull request #6102: Build: Fix build version

2022-11-10 Thread GitBox
ajantha-bhat closed pull request #6102: Build: Fix build version URL: https://github.com/apache/iceberg/pull/6102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[GitHub] [iceberg] luoyuxia commented on issue #6104: Rewrite iceberg small files with flink succeeds but no snapshot is generated (V2 - upsert model)

2022-11-10 Thread GitBox
luoyuxia commented on issue #6104: URL: https://github.com/apache/iceberg/issues/6104#issuecomment-1310193327 > the repeated compression process of full data The compaction won't be repeated. I mean during one compaction, some files are compacted to one file. In next compaction, it won't

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6074: API,Core: SnapshotManager to be created through Transaction

2022-11-10 Thread GitBox
gaborkaszab commented on code in PR #6074: URL: https://github.com/apache/iceberg/pull/6074#discussion_r1019036196 ## core/src/main/java/org/apache/iceberg/SnapshotManager.java: ## @@ -30,6 +31,17 @@ public class SnapshotManager implements ManageSnapshots { ops.current(

[GitHub] [iceberg] nastra commented on a diff in pull request #6168: Build: Bump jackson from 2.11.4 to 2.14.0

2022-11-10 Thread GitBox
nastra commented on code in PR #6168: URL: https://github.com/apache/iceberg/pull/6168#discussion_r1019154280 ## spark/v3.2/build.gradle: ## @@ -33,6 +33,8 @@ configure(sparkProjects) { resolutionStrategy { force "com.fasterxml.jackson.module:jackson-module-scal

[GitHub] [iceberg] nastra commented on a diff in pull request #6168: Build: Bump jackson from 2.11.4 to 2.14.0

2022-11-10 Thread GitBox
nastra commented on code in PR #6168: URL: https://github.com/apache/iceberg/pull/6168#discussion_r1019154927 ## spark/v3.3/build.gradle: ## @@ -31,7 +31,9 @@ configure(sparkProjects) { configurations { all { resolutionStrategy { +force "com.fasterxml.jac

[GitHub] [iceberg] nastra opened a new pull request, #6169: AWS,Core: Add S3 Signer client + REST Spec

2022-11-10 Thread GitBox
nastra opened a new pull request, #6169: URL: https://github.com/apache/iceberg/pull/6169 This introduces an S3 REST signer client and defines a REST spec (`s3-signer-open-api.yml`) for a server implementation. Below is a high-level overview of the introduced changes: * the main logi

[GitHub] [iceberg] nastra commented on pull request #6169: AWS,Core: Add S3 REST Signer client + REST Spec

2022-11-10 Thread GitBox
nastra commented on PR #6169: URL: https://github.com/apache/iceberg/pull/6169#issuecomment-1310373071 /cc @rdblue @danielcweeks @jackye1995 @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [iceberg] nastra commented on pull request #4855: Infra: add AWS CI workflow configuration

2022-11-10 Thread GitBox
nastra commented on PR #4855: URL: https://github.com/apache/iceberg/pull/4855#issuecomment-1310414693 @jackye1995 any updates on this? I think it would be great to get this in -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [iceberg] nastra commented on pull request #6169: AWS,Core: Add S3 REST Signer client + REST Spec

2022-11-10 Thread GitBox
nastra commented on PR #6169: URL: https://github.com/apache/iceberg/pull/6169#issuecomment-1310416636 For testing we might actually need https://github.com/apache/iceberg/pull/4855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [iceberg] Fokko opened a new pull request, #6170: Python: Move FileIO initialization to the catalog

2022-11-10 Thread GitBox
Fokko opened a new pull request, #6170: URL: https://github.com/apache/iceberg/pull/6170 Follow up on: https://github.com/apache/iceberg/pull/6161#discussion_r1018547445 Another option is to move it to the constructor of the Table and not store the properties. -- This is an automa

[GitHub] [iceberg] Fokko commented on a diff in pull request #6161: Python: Load the S3 Session token

2022-11-10 Thread GitBox
Fokko commented on code in PR #6161: URL: https://github.com/apache/iceberg/pull/6161#discussion_r1019271623 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +110,16 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntry

[GitHub] [iceberg] Fokko commented on a diff in pull request #6161: Python: Load the S3 Session token

2022-11-10 Thread GitBox
Fokko commented on code in PR #6161: URL: https://github.com/apache/iceberg/pull/6161#discussion_r1019271623 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +110,16 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntry

[GitHub] [iceberg] thompson0012 opened a new issue, #6171: iceberg cant read parquet after configuration

2022-11-10 Thread GitBox
thompson0012 opened a new issue, #6171: URL: https://github.com/apache/iceberg/issues/6171 ### Apache Iceberg version 1.0.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 as the official docker image has several problem i use the py

[GitHub] [iceberg] nastra commented on issue #6171: iceberg cant read parquet after configuration

2022-11-10 Thread GitBox
nastra commented on issue #6171: URL: https://github.com/apache/iceberg/issues/6171#issuecomment-1310475564 `NoClassDefFoundError` usually indicates that some classes were available during compilation but are not available anymore at runtime. Also what issue does the Docker image have? Wo

[GitHub] [iceberg] gaborkaszab commented on issue #6042: Add delete file information to partitions table

2022-11-10 Thread GitBox
gaborkaszab commented on issue #6042: URL: https://github.com/apache/iceberg/issues/6042#issuecomment-1310484363 Well, my plan was to keep the existing columns in the partitions metadata table intact and add some delete file related ones on top. Currently, I only dealt with position deletes

[GitHub] [iceberg] thompson0012 commented on issue #6171: iceberg cant read parquet after configuration

2022-11-10 Thread GitBox
thompson0012 commented on issue #6171: URL: https://github.com/apache/iceberg/issues/6171#issuecomment-1310487799 i know there are some issues found in #5993 , so i use this [images](https://hub.docker.com/r/jupyter/pyspark-notebook) with the following .jars 1. [1.0.0 Spark 3.3_2.12 ru

[GitHub] [iceberg] Fokko commented on pull request #6145: Python: Add initial TableScan implementation

2022-11-10 Thread GitBox
Fokko commented on PR #6145: URL: https://github.com/apache/iceberg/pull/6145#issuecomment-1310545882 Also threw in some tests πŸ‘πŸ» -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [iceberg] thompson0012 commented on issue #6171: iceberg cant read parquet after configuration

2022-11-10 Thread GitBox
thompson0012 commented on issue #6171: URL: https://github.com/apache/iceberg/issues/6171#issuecomment-1310555881 i just found that, almost all the read extension is fail when i added iceberg configuration i cant normally read my local files after setting up the iceberg but it goes

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019379366 ## python/pyiceberg/expressions/literals.py: ## @@ -58,25 +60,25 @@ timestamp_to_micros, timestamptz_to_micros, ) -from pyiceberg.utils.singleton import Singl

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019381524 ## python/pyiceberg/expressions/literals.py: ## @@ -108,7 +110,7 @@ def __ge__(self, other): @singledispatch -def literal(value) -> Literal: +def literal(value: Any

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019382186 ## python/pyiceberg/expressions/literals.py: ## @@ -125,81 +127,73 @@ def literal(value) -> Literal: @literal.register(bool) -def _(value: bool) -> Literal[bool]: +

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019382620 ## python/pyiceberg/expressions/literals.py: ## @@ -125,81 +127,71 @@ def literal(value) -> Literal: @literal.register(bool) -def _(value: bool) -> Literal[bool]: +

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019384042 ## python/pyiceberg/expressions/literals.py: ## @@ -125,81 +127,73 @@ def literal(value) -> Literal: @literal.register(bool) -def _(value: bool) -> Literal[bool]: +

[GitHub] [iceberg] rdblue commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
rdblue commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019385197 ## python/pyiceberg/expressions/literals.py: ## @@ -309,23 +303,23 @@ def __init__(self, value: float): super().__init__(value, float) @singledispatchmet

[GitHub] [iceberg] Fokko commented on a diff in pull request #6141: Python: Make invalid Literal conversions explicit

2022-11-10 Thread GitBox
Fokko commented on code in PR #6141: URL: https://github.com/apache/iceberg/pull/6141#discussion_r1019389663 ## python/pyiceberg/expressions/literals.py: ## @@ -108,7 +110,7 @@ def __ge__(self, other): @singledispatch -def literal(value) -> Literal: +def literal(value: Any)

[GitHub] [iceberg] rdblue merged pull request #6145: Python: Add initial TableScan implementation

2022-11-10 Thread GitBox
rdblue merged PR #6145: URL: https://github.com/apache/iceberg/pull/6145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue closed pull request #6131: Python: Add initial TableScan implementation

2022-11-10 Thread GitBox
rdblue closed pull request #6131: Python: Add initial TableScan implementation URL: https://github.com/apache/iceberg/pull/6131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [iceberg] rdblue commented on pull request #6131: Python: Add initial TableScan implementation

2022-11-10 Thread GitBox
rdblue commented on PR #6131: URL: https://github.com/apache/iceberg/pull/6131#issuecomment-1310623432 Closing in favor of #6145. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [iceberg] rdblue commented on pull request #6145: Python: Add initial TableScan implementation

2022-11-10 Thread GitBox
rdblue commented on PR #6145: URL: https://github.com/apache/iceberg/pull/6145#issuecomment-1310622915 Thanks, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [iceberg] rdblue commented on pull request #6168: Build: Bump jackson from 2.11.4 to 2.14.0

2022-11-10 Thread GitBox
rdblue commented on PR #6168: URL: https://github.com/apache/iceberg/pull/6168#issuecomment-1310624837 We should probably wait on this one until after the 1.1 release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [iceberg] rdblue merged pull request #6167: Build: Add -DallVersions property that exposes all component versions

2022-11-10 Thread GitBox
rdblue merged PR #6167: URL: https://github.com/apache/iceberg/pull/6167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6167: Build: Add -DallVersions property that exposes all component versions

2022-11-10 Thread GitBox
rdblue commented on PR #6167: URL: https://github.com/apache/iceberg/pull/6167#issuecomment-1310626722 Thanks, @nastra! This is going to make it much easier to keep test running properly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [iceberg] rdblue commented on a diff in pull request #6170: Python: Move FileIO initialization to the catalog

2022-11-10 Thread GitBox
rdblue commented on code in PR #6170: URL: https://github.com/apache/iceberg/pull/6170#discussion_r1019421535 ## python/pyiceberg/catalog/__init__.py: ## @@ -162,6 +163,9 @@ def __init__(self, name: str, **properties: str): self.name = name self.properties = pr

[GitHub] [iceberg] rdblue commented on pull request #6170: Python: Move FileIO initialization to the catalog

2022-11-10 Thread GitBox
rdblue commented on PR #6170: URL: https://github.com/apache/iceberg/pull/6170#issuecomment-1310637803 Looks good to me! Just needs to be rebased and we can get it in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

<    12   13   14   15   16   17   18   19   20   21   >