[GitHub] [iceberg] Fokko commented on a diff in pull request #6131: Python: Add initial TableScan implementation

2022-11-08 Thread GitBox
Fokko commented on code in PR #6131: URL: https://github.com/apache/iceberg/pull/6131#discussion_r1016261138 ## python/pyiceberg/table/__init__.py: ## @@ -14,30 +14,43 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and

[GitHub] [iceberg] 0xffmeta commented on issue #2686: Read Iceberg Table via Hive: Hadoop Tables Succeed. Hive Catalog Failed. Anyone met the errors before?

2022-11-08 Thread GitBox
0xffmeta commented on issue #2686: URL: https://github.com/apache/iceberg/issues/2686#issuecomment-1306804533 Thanks @luoyuxia. It's working now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] pvary commented on issue #6071: Should ClientPool consider UGI when reusing a connection?

2022-11-08 Thread GitBox
pvary commented on issue #6071: URL: https://github.com/apache/iceberg/issues/6071#issuecomment-1306825222 @lirui-apache: you are correct. Even though the Catalogs should not be shared between sessions, we might end up sharing pools in the same JVM -- This is an automated message from the

[GitHub] [iceberg] nastra opened a new issue, #6144: Enable revapi on iceberg-core

2022-11-08 Thread GitBox
nastra opened a new issue, #6144: URL: https://github.com/apache/iceberg/issues/6144 ### Feature Request / Improvement I enabled `revapi` on the `iceberg-core` module to check its current state in terms of API/ABI compatibility with 1.0.0. Below is the result: ``` > Task

[GitHub] [iceberg] Fokko opened a new pull request, #6145: Python: Add initial TableScan implementation

2022-11-08 Thread GitBox
Fokko opened a new pull request, #6145: URL: https://github.com/apache/iceberg/pull/6145 This adds an implementation of TableScan that is an alternative to the one in https://github.com/apache/iceberg/pull/6131. This doesn't implement plan_files, it is just to demonstrate a possible scan AP

[GitHub] [iceberg] Fokko commented on issue #6142: Provide a binary installing using the latest Master code in Python

2022-11-08 Thread GitBox
Fokko commented on issue #6142: URL: https://github.com/apache/iceberg/issues/6142#issuecomment-1306858941 Hey @djouallah thanks for reaching out. You can install it directly from git. Please try: ``` pip install "git+https://github.com/apache/iceberg.git#egg=pyiceberg&subdirectory=py

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
gaborkaszab commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1015406783 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -518,11 +522,36 @@ private Map convertToMetadata(Database database) { if (data

[GitHub] [iceberg] lirui-apache commented on issue #6071: Should ClientPool consider UGI when reusing a connection?

2022-11-08 Thread GitBox
lirui-apache commented on issue #6071: URL: https://github.com/apache/iceberg/issues/6071#issuecomment-1306909031 > Even though the Catalogs should not be shared between sessions, we might end up sharing pools in the same JVM Thanks @pvary. Is this the intended behavior at the moment?

[GitHub] [iceberg] djouallah commented on issue #6142: Provide a binary installing using the latest Master code in Python

2022-11-08 Thread GitBox
djouallah commented on issue #6142: URL: https://github.com/apache/iceberg/issues/6142#issuecomment-1306918536 huh, git what is git ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] ajantha-bhat commented on pull request #6053: Build: Let revapi compare API compatibility against apache-iceberg-1.0.0

2022-11-08 Thread GitBox
ajantha-bhat commented on PR #6053: URL: https://github.com/apache/iceberg/pull/6053#issuecomment-1307017093 > Let's change this to the below diff then. That should work. I don't know why the same version definition doesn't work inside revapi.yml: Locally, I have changed and added new

[GitHub] [iceberg] hililiwei commented on pull request #6088: HuaweiCloud: Introduce the iceberg-huaweicloud

2022-11-08 Thread GitBox
hililiwei commented on PR #6088: URL: https://github.com/apache/iceberg/pull/6088#issuecomment-1307078172 cc @rdblue , could you please take a look when you have a chance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [iceberg] nastra commented on pull request #6053: Build: Let revapi compare API compatibility against apache-iceberg-1.0.0

2022-11-08 Thread GitBox
nastra commented on PR #6053: URL: https://github.com/apache/iceberg/pull/6053#issuecomment-1307096836 @rdblue can you review this one please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [iceberg] pvary commented on issue #6071: Should ClientPool consider UGI when reusing a connection?

2022-11-08 Thread GitBox
pvary commented on issue #6071: URL: https://github.com/apache/iceberg/issues/6071#issuecomment-1307097033 As you have a usecase which needs this now, and even at the time of the decision it was clear that it would be needed in the longer term, I would be happy to review the PR if you have

[GitHub] [iceberg] ajantha-bhat commented on pull request #6053: Build: Let revapi compare API compatibility against apache-iceberg-1.0.0

2022-11-08 Thread GitBox
ajantha-bhat commented on PR #6053: URL: https://github.com/apache/iceberg/pull/6053#issuecomment-1307099782 > Locally, I have changed and added new interface to test it out. Sadly, Interface change was not caught by revAPI. So, this might not be the right fix. the latest update,

[GitHub] [iceberg] luoyuxia commented on issue #2653: Flink: Can not see hive udf in Flink when use iceberg hive catalog

2022-11-08 Thread GitBox
luoyuxia commented on issue #2653: URL: https://github.com/apache/iceberg/issues/2653#issuecomment-1307155383 I don't think it should be supported in iceberg , it's too specific and many other catalog implementation will never have udf. For you case, I'll still advice you to use the hive

[GitHub] [iceberg] luoyuxia commented on issue #2839: iceberg.flink.TableLoader from hive , can't resolve hdfs nameservice

2022-11-08 Thread GitBox
luoyuxia commented on issue #2839: URL: https://github.com/apache/iceberg/issues/2839#issuecomment-1307160793 Is the network fine from your IDEA to the hdfs cluster? Can the host `nsdev ` be resolved correctly? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [iceberg] luoyuxia commented on issue #1995: Flink : table column comment missing

2022-11-08 Thread GitBox
luoyuxia commented on issue #1995: URL: https://github.com/apache/iceberg/issues/1995#issuecomment-1307188822 Flink itself dosen't support column comment. There is [FLINK-22936)](https://issues.apache.org/jira/browse/FLINK-22936) for supporting it. -- This is an automated message from

[GitHub] [iceberg] Fokko merged pull request #6094: Spark-3.0: Remove spark/v3.0 folder

2022-11-08 Thread GitBox
Fokko merged PR #6094: URL: https://github.com/apache/iceberg/pull/6094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6094: Spark-3.0: Remove spark/v3.0 folder

2022-11-08 Thread GitBox
Fokko commented on PR #6094: URL: https://github.com/apache/iceberg/pull/6094#issuecomment-1307258087 Thanks @ajantha-bhat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [iceberg] Fokko opened a new pull request, #6147: Python: Pin mypy

2022-11-08 Thread GitBox
Fokko opened a new pull request, #6147: URL: https://github.com/apache/iceberg/pull/6147 The CI is failing because a new version of mypy has been released yesterday: https://pypi.org/project/mypy/0.982/ -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [iceberg] hililiwei closed pull request #4969: Flink 1.15: include overwrite operation when read incremental data

2022-11-08 Thread GitBox
hililiwei closed pull request #4969: Flink 1.15: include overwrite operation when read incremental data URL: https://github.com/apache/iceberg/pull/4969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [iceberg] hililiwei commented on pull request #3991: Flink: Support nested projection

2022-11-08 Thread GitBox
hililiwei commented on PR #3991: URL: https://github.com/apache/iceberg/pull/3991#issuecomment-1307309768 cc @stevenzwu @Fokko @chenjunjiedada @singhpk234 , could you please take a look when you get a chance? -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [iceberg] Fokko opened a new issue, #6148: PyIceberg: Bump mypy to v0.990

2022-11-08 Thread GitBox
Fokko opened a new issue, #6148: URL: https://github.com/apache/iceberg/issues/6148 ### Feature Request / Improvement Bump mypy to the latest version. This one has support for recursive types: https://mypy-lang.blogspot.com/2022/11/mypy-0990-released.html This is nice si

[GitHub] [iceberg] szehon-ho merged pull request #4577: Fixes read metadata table failed due to illegal character

2022-11-08 Thread GitBox
szehon-ho merged PR #4577: URL: https://github.com/apache/iceberg/pull/4577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg] szehon-ho commented on pull request #4577: Fixes read metadata table failed due to illegal character

2022-11-08 Thread GitBox
szehon-ho commented on PR #4577: URL: https://github.com/apache/iceberg/pull/4577#issuecomment-1307316351 Merged, thanks @ConeyLiu , and also @nastra @chenjunjiedada @RussellSpitzer for reviews -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [iceberg] wobu commented on issue #3838: Spark V3.2 : spark-sql extension stored procedure remove_orphan_files fails with S3 IO implementation

2022-11-08 Thread GitBox
wobu commented on issue #3838: URL: https://github.com/apache/iceberg/issues/3838#issuecomment-1307325556 still happens with Iceberg 1.0.0 and spark runtime 3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6147: Python: Pin mypy in Python Legacy

2022-11-08 Thread GitBox
ajantha-bhat commented on code in PR #6147: URL: https://github.com/apache/iceberg/pull/6147#discussion_r1016727344 ## python_legacy/tox.ini: ## @@ -59,7 +59,7 @@ commands = basepython = python3 skip_install = true deps = -mypy +mypy==0.982 Review Comment: nit: ma

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6111: Flink: Add 'cache.expiration-interval-ms' option to FlinkCatalog

2022-11-08 Thread GitBox
stevenzwu commented on code in PR #6111: URL: https://github.com/apache/iceberg/pull/6111#discussion_r1016788395 ## flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -145,8 +145,27 @@ protected Catalog createCatalog( baseNamespace =

[GitHub] [iceberg] nastra commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common modules

2022-11-08 Thread GitBox
nastra commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1016820044 ## build.gradle: ## @@ -91,6 +91,7 @@ dependencyRecommendations { } def projectVersion = getProjectVersion() +final REVAPI_PREOJECTS = ["iceberg-api", "iceberg-core"

[GitHub] [iceberg] rdblue commented on pull request #6073: Core: Pass purgeRequested flag to REST server

2022-11-08 Thread GitBox
rdblue commented on PR #6073: URL: https://github.com/apache/iceberg/pull/6073#issuecomment-1307507487 Looks good! Thanks for the updates, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] rdblue merged pull request #6073: Core: Pass purgeRequested flag to REST server

2022-11-08 Thread GitBox
rdblue merged PR #6073: URL: https://github.com/apache/iceberg/pull/6073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1016924294 ## api/src/main/java/org/apache/iceberg/FileScanTask.java: ## @@ -21,7 +21,8 @@ import java.util.List; /** A scan task over a range of bytes in a single data file.

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1016924924 ## api/src/main/java/org/apache/iceberg/util/StructProjection.java: ## @@ -90,6 +91,19 @@ public static StructProjection createAllowMissing( private final StructProj

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1016925588 ## api/src/main/java/org/apache/iceberg/util/StructProjection.java: ## @@ -171,6 +185,14 @@ public StructProjection wrap(StructLike newStruct) { return this; }

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
haizhou-zhao commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1016941186 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -518,11 +522,36 @@ private Map convertToMetadata(Database database) { if (dat

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
haizhou-zhao commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1016941614 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -518,11 +522,36 @@ private Map convertToMetadata(Database database) { if (dat

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
haizhou-zhao commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1016943632 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -358,6 +359,63 @@ public void testCreateNamespace() throws Exception {

[GitHub] [iceberg] Fokko commented on a diff in pull request #6140: Python: Fix Evaluator tests

2022-11-08 Thread GitBox
Fokko commented on code in PR #6140: URL: https://github.com/apache/iceberg/pull/6140#discussion_r1016944001 ## python/tests/expressions/test_visitors.py: ## @@ -827,24 +834,23 @@ def _to_manifest_file(*partitions: PartitionFieldSummary) -> ManifestFile: ) -def _create

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
haizhou-zhao commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1016944554 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -448,6 +575,59 @@ public void testRemoveNamespaceProperties() throws TExceptio

[GitHub] [iceberg] haizhou-zhao commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
haizhou-zhao commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1016947075 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -448,6 +540,36 @@ public void testRemoveNamespaceProperties() throws TExceptio

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1016948823 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -128,6 +137,66 @@ public static CloseableIterable> planTaskG combinedTasks -> new Bas

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1016952512 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -128,6 +137,66 @@ public static CloseableIterable> planTaskG combinedTasks -> new Bas

[GitHub] [iceberg] danielcweeks merged pull request #6147: Python: Pin mypy in Python Legacy

2022-11-08 Thread GitBox
danielcweeks merged PR #6147: URL: https://github.com/apache/iceberg/pull/6147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
aokolnychyi commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1017019275 ## api/src/main/java/org/apache/iceberg/util/StructProjection.java: ## @@ -90,6 +91,19 @@ public static StructProjection createAllowMissing( private final Struct

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
aokolnychyi commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1017020894 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -128,6 +137,66 @@ public static CloseableIterable> planTaskG combinedTasks -> new

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #5376: Core: Add readable metrics columns to files metadata tables

2022-11-08 Thread GitBox
RussellSpitzer commented on code in PR #5376: URL: https://github.com/apache/iceberg/pull/5376#discussion_r1017037292 ## core/src/main/java/org/apache/iceberg/BaseFilesTable.java: ## @@ -140,42 +142,75 @@ protected CloseableIterable doPlanFiles() { } static class Manifes

[GitHub] [iceberg] jzhuge commented on a diff in pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
jzhuge commented on code in PR #4925: URL: https://github.com/apache/iceberg/pull/4925#discussion_r1017039590 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] jzhuge commented on a diff in pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
jzhuge commented on code in PR #4925: URL: https://github.com/apache/iceberg/pull/4925#discussion_r1017042399 ## api/src/main/java/org/apache/iceberg/view/SQLViewRepresentation.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] Fokko commented on pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
Fokko commented on PR #6139: URL: https://github.com/apache/iceberg/pull/6139#issuecomment-1307812458 @rdblue added support for wrapping `str` automatically into a `Reference()`. I did notice some funky behavior with the type system. When using a `Reference(UnboundTerm[T])` mypy it is unabl

[GitHub] [iceberg] Fokko commented on a diff in pull request #6146: Build: Enable revapi on core/parquet/orc/common/data modules

2022-11-08 Thread GitBox
Fokko commented on code in PR #6146: URL: https://github.com/apache/iceberg/pull/6146#discussion_r1017125840 ## build.gradle: ## @@ -91,6 +91,7 @@ dependencyRecommendations { } def projectVersion = getProjectVersion() +final REVAPI_PREOJECTS = ["iceberg-api", "iceberg-core",

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6045: [iceberg-hive-metastore] Support setting individual and group ownership for Namespace

2022-11-08 Thread GitBox
danielcweeks commented on code in PR #6045: URL: https://github.com/apache/iceberg/pull/6045#discussion_r1017130690 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -360,5 +360,7 @@ private TableProperties() {} public static final String UPSERT_ENABLED = "

[GitHub] [iceberg] jzhuge commented on pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
jzhuge commented on PR #4925: URL: https://github.com/apache/iceberg/pull/4925#issuecomment-1307936107 > > * No need for something like `View.updateRepresentions()`, as `buildView() + replace()` should be enough > > Can one replace the current version or an old version by adding a new

[GitHub] [iceberg] wmoustafa commented on pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
wmoustafa commented on PR #4925: URL: https://github.com/apache/iceberg/pull/4925#issuecomment-1307939119 `buildView() + withOtherSQLRepresentation() + replace()` automatically copies existing dialects from previous versions or if we want to preserve them, we need to add them again? -- T

[GitHub] [iceberg] rbalamohan opened a new issue, #6149: Consider parquet.writer.version=2 for positional delete files

2022-11-08 Thread GitBox
rbalamohan opened a new issue, #6149: URL: https://github.com/apache/iceberg/issues/6149 ### Apache Iceberg version 0.14.1 ### Query engine Spark ### Please describe the bug 🐞 Positional delete files contain the file position and the filename. Currently pos

[GitHub] [iceberg] jzhuge commented on pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
jzhuge commented on PR #4925: URL: https://github.com/apache/iceberg/pull/4925#issuecomment-1307975405 > `buildView() + withOtherSQLRepresentation() + replace()` automatically copies existing dialects from previous versions or if we want to preserve them, we need to add them again? `

[GitHub] [iceberg] jzhuge commented on a diff in pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
jzhuge commented on code in PR #4925: URL: https://github.com/apache/iceberg/pull/4925#discussion_r1017221086 ## api/src/main/java/org/apache/iceberg/view/ViewVersion.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

[GitHub] [iceberg] rdblue merged pull request #6053: Build: Let revapi compare API compatibility against apache-iceberg-1.0.0

2022-11-08 Thread GitBox
rdblue merged PR #6053: URL: https://github.com/apache/iceberg/pull/6053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6053: Build: Let revapi compare API compatibility against apache-iceberg-1.0.0

2022-11-08 Thread GitBox
rdblue commented on PR #6053: URL: https://github.com/apache/iceberg/pull/6053#issuecomment-1307993603 Looks good to me! Thanks, @ajantha-bhat! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [iceberg] wmoustafa commented on a diff in pull request #4925: API: Add view interfaces

2022-11-08 Thread GitBox
wmoustafa commented on code in PR #4925: URL: https://github.com/apache/iceberg/pull/4925#discussion_r1017235634 ## api/src/main/java/org/apache/iceberg/view/ViewBuilder.java: ## @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017237937 ## python/pyiceberg/expressions/__init__.py: ## @@ -32,6 +41,24 @@ B = TypeVar("B") +def _to_literal(lit: Optional[Union[T, Literal[T]]]) -> Optional[Literal[T]]: +

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017240440 ## python/tests/expressions/test_expressions.py: ## @@ -272,6 +235,22 @@ def test_in_empty(): assert In(Reference("foo"), ()) == AlwaysFalse() +def test_in_set(

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017240637 ## python/tests/expressions/test_expressions.py: ## @@ -272,6 +235,22 @@ def test_in_empty(): assert In(Reference("foo"), ()) == AlwaysFalse() +def test_in_set(

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017240780 ## python/tests/expressions/test_expressions.py: ## @@ -281,15 +260,15 @@ def test_not_in_equal(): def test_bind_in(table_schema_simple: Schema): -bound = Bound

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017242654 ## python/tests/expressions/test_visitors.py: ## @@ -284,13 +277,14 @@ def test_boolean_expression_visit_raise_not_implemented_error(): def test_bind_visitor_alrea

[GitHub] [iceberg] rdblue commented on a diff in pull request #6139: Python: Remove dataclasses

2022-11-08 Thread GitBox
rdblue commented on code in PR #6139: URL: https://github.com/apache/iceberg/pull/6139#discussion_r1017243171 ## python/tests/expressions/test_expressions.py: ## @@ -365,7 +344,7 @@ def test_bound_greater_than_or_equal_invert(table_schema_simple: Schema): def test_bound_gr

[GitHub] [iceberg] rdblue commented on a diff in pull request #6140: Python: Fix Evaluator tests

2022-11-08 Thread GitBox
rdblue commented on code in PR #6140: URL: https://github.com/apache/iceberg/pull/6140#discussion_r1017243615 ## python/tests/expressions/test_visitors.py: ## @@ -1614,27 +1342,14 @@ def test_manifest_evaluator_and(): upper_bound=_to_byte_buffer(LongType(), 10),

[GitHub] [iceberg] rdblue commented on a diff in pull request #6140: Python: Fix Evaluator tests

2022-11-08 Thread GitBox
rdblue commented on code in PR #6140: URL: https://github.com/apache/iceberg/pull/6140#discussion_r1017243851 ## python/tests/expressions/test_visitors.py: ## @@ -1585,27 +1326,14 @@ def test_manifest_evaluator_not(): upper_bound=_to_byte_buffer(LongType(), 10),

[GitHub] [iceberg] rdblue commented on a diff in pull request #6140: Python: Fix Evaluator tests

2022-11-08 Thread GitBox
rdblue commented on code in PR #6140: URL: https://github.com/apache/iceberg/pull/6140#discussion_r1017244665 ## python/tests/expressions/test_visitors.py: ## @@ -1039,36 +985,37 @@ def test_manifest_evaluator_equal_overlap(): ) ], ) - -assert _cre

[GitHub] [iceberg] rdblue commented on a diff in pull request #6131: Python: Add initial TableScan implementation

2022-11-08 Thread GitBox
rdblue commented on code in PR #6131: URL: https://github.com/apache/iceberg/pull/6131#discussion_r1017248707 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +103,90 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntr

[GitHub] [iceberg] rdblue commented on a diff in pull request #6145: Python: Add initial TableScan implementation

2022-11-08 Thread GitBox
rdblue commented on code in PR #6145: URL: https://github.com/apache/iceberg/pull/6145#discussion_r1017249517 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +119,53 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntr

[GitHub] [iceberg] rdblue commented on a diff in pull request #6145: Python: Add initial TableScan implementation

2022-11-08 Thread GitBox
rdblue commented on code in PR #6145: URL: https://github.com/apache/iceberg/pull/6145#discussion_r1017251652 ## python/pyiceberg/table/__init__.py: ## @@ -90,3 +119,53 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: def history(self) -> List[SnapshotLogEntr

[GitHub] [iceberg] github-actions[bot] commented on issue #4661: Metadata table queries fail if a partition column was reused in V2

2022-11-08 Thread GitBox
github-actions[bot] commented on issue #4661: URL: https://github.com/apache/iceberg/issues/4661#issuecomment-1308011931 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] commented on issue #4647: Snapshot summary should contain default values

2022-11-08 Thread GitBox
github-actions[bot] commented on issue #4647: URL: https://github.com/apache/iceberg/issues/4647#issuecomment-1308011953 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] closed issue #4661: Metadata table queries fail if a partition column was reused in V2

2022-11-08 Thread GitBox
github-actions[bot] closed issue #4661: Metadata table queries fail if a partition column was reused in V2 URL: https://github.com/apache/iceberg/issues/4661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [iceberg] github-actions[bot] closed issue #4647: Snapshot summary should contain default values

2022-11-08 Thread GitBox
github-actions[bot] closed issue #4647: Snapshot summary should contain default values URL: https://github.com/apache/iceberg/issues/4647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] github-actions[bot] commented on issue #4639: A problem occurs when updating/deleting with spark-sql after loading data with iceberg Java api

2022-11-08 Thread GitBox
github-actions[bot] commented on issue #4639: URL: https://github.com/apache/iceberg/issues/4639#issuecomment-1308011978 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] closed issue #4639: A problem occurs when updating/deleting with spark-sql after loading data with iceberg Java api

2022-11-08 Thread GitBox
github-actions[bot] closed issue #4639: A problem occurs when updating/deleting with spark-sql after loading data with iceberg Java api URL: https://github.com/apache/iceberg/issues/4639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [iceberg] github-actions[bot] commented on issue #4628: missing SetWriteDistributionAndOrdering class for spark sql plan

2022-11-08 Thread GitBox
github-actions[bot] commented on issue #4628: URL: https://github.com/apache/iceberg/issues/4628#issuecomment-1308012010 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] github-actions[bot] commented on issue #4563: `ALTER TABLE ... DROP COLUMN` allows dropping a column used by old PartitionSpecs

2022-11-08 Thread GitBox
github-actions[bot] commented on issue #4563: URL: https://github.com/apache/iceberg/issues/4563#issuecomment-1308012053 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] closed issue #4628: missing SetWriteDistributionAndOrdering class for spark sql plan

2022-11-08 Thread GitBox
github-actions[bot] closed issue #4628: missing SetWriteDistributionAndOrdering class for spark sql plan URL: https://github.com/apache/iceberg/issues/4628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] rdblue opened a new pull request, #6150: Core: Sync client/server properties in REST catalog

2022-11-08 Thread GitBox
rdblue opened a new pull request, #6150: URL: https://github.com/apache/iceberg/pull/6150 This PR adds query params to the REST config route that are used to sync configuration between the REST client and service. Right now, this sends the only client catalog property that needs to be

[GitHub] [iceberg] lvyanquan commented on a diff in pull request #6111: Flink: Add 'cache.expiration-interval-ms' option to FlinkCatalog

2022-11-08 Thread GitBox
lvyanquan commented on code in PR #6111: URL: https://github.com/apache/iceberg/pull/6111#discussion_r1017299727 ## flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -145,8 +145,27 @@ protected Catalog createCatalog( baseNamespace =

[GitHub] [iceberg] lvyanquan commented on a diff in pull request #6111: Flink: Add 'cache.expiration-interval-ms' option to FlinkCatalog

2022-11-08 Thread GitBox
lvyanquan commented on code in PR #6111: URL: https://github.com/apache/iceberg/pull/6111#discussion_r1017299727 ## flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -145,8 +145,27 @@ protected Catalog createCatalog( baseNamespace =

[GitHub] [iceberg] ConeyLiu commented on pull request #4577: Fixes read metadata table failed due to illegal character

2022-11-08 Thread GitBox
ConeyLiu commented on PR #4577: URL: https://github.com/apache/iceberg/pull/4577#issuecomment-1308090077 Thanks @szehon-ho @nastra @chenjunjiedada @RussellSpitzer for your time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] ConeyLiu closed issue #4576: Read metadata table failed due to illegal character

2022-11-08 Thread GitBox
ConeyLiu closed issue #4576: Read metadata table failed due to illegal character URL: https://github.com/apache/iceberg/issues/4576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6111: Flink: Add 'cache.expiration-interval-ms' option to FlinkCatalog

2022-11-08 Thread GitBox
stevenzwu commented on code in PR #6111: URL: https://github.com/apache/iceberg/pull/6111#discussion_r1017328011 ## flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -145,8 +145,27 @@ protected Catalog createCatalog( baseNamespace =

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1017341461 ## core/src/test/java/org/apache/iceberg/util/TestTableScanUtil.java: ## @@ -136,6 +141,129 @@ public void testTaskMerging() { Assert.assertEquals("Appropriate tas

[GitHub] [iceberg] sunchao commented on a diff in pull request #2276: Core: Add option to combine tasks by partition

2022-11-08 Thread GitBox
sunchao commented on code in PR #2276: URL: https://github.com/apache/iceberg/pull/2276#discussion_r1017342018 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -128,6 +137,66 @@ public static CloseableIterable> planTaskG combinedTasks -> new Bas

[GitHub] [iceberg-docs] ajantha-bhat opened a new pull request, #175: Docs: Update spark-3.0 removal

2022-11-08 Thread GitBox
ajantha-bhat opened a new pull request, #175: URL: https://github.com/apache/iceberg-docs/pull/175 Follow up of https://github.com/apache/iceberg/pull/6093 for docs repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg-docs] ajantha-bhat commented on pull request #175: Docs: Update spark-3.0 removal

2022-11-08 Thread GitBox
ajantha-bhat commented on PR #175: URL: https://github.com/apache/iceberg-docs/pull/175#issuecomment-1308117214 cc: @hililiwei, we might have to do the same for flink 1.13 removal and 1.16 addition. -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [iceberg] lirui-apache commented on issue #6071: Should ClientPool consider UGI when reusing a connection?

2022-11-08 Thread GitBox
lirui-apache commented on issue #6071: URL: https://github.com/apache/iceberg/issues/6071#issuecomment-1308151509 OK, I'll work on a PR for this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6151: Docs: Update table snapshot retention property descriptions

2022-11-08 Thread GitBox
amogh-jahagirdar opened a new pull request, #6151: URL: https://github.com/apache/iceberg/pull/6151 This PR updates the table snapshot retention property descriptions so that they explicitly mention they control the min snapshots to keep and max age of snapshots on table's main branch. -

[GitHub] [iceberg] amogh-jahagirdar closed pull request #6151: Docs: Update table snapshot retention property descriptions

2022-11-08 Thread GitBox
amogh-jahagirdar closed pull request #6151: Docs: Update table snapshot retention property descriptions URL: https://github.com/apache/iceberg/pull/6151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #6151: Docs: Update table snapshot retention property descriptions

2022-11-08 Thread GitBox
amogh-jahagirdar commented on PR #6151: URL: https://github.com/apache/iceberg/pull/6151#issuecomment-1308203757 closing this to avoid confusion, and will raise another one. This change should basically be the opposite. This configuration gets used by default for any branch not just main.

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6152: Docs: Update table snapshot retention property descriptions

2022-11-08 Thread GitBox
amogh-jahagirdar opened a new pull request, #6152: URL: https://github.com/apache/iceberg/pull/6152 Docs: Update table snapshot retention property descriptions to explicitly mention that it is a default for all the table's branches. -- This is an automated message from the Apache Git Serv

[GitHub] [iceberg] zhangpengbigdata opened a new issue, #6153: I found duplicate records when i was repeatedly exporting records from CDC Stream into iceberg partitioned table

2022-11-08 Thread GitBox
zhangpengbigdata opened a new issue, #6153: URL: https://github.com/apache/iceberg/issues/6153 ### Query engine Iceberg 1.0.0 Flink1.13 ### Question Hi all, I found duplicate records when i was repeatedly exporting records from CDC Stream into iceberg partitioned tab