Re: [I] Move `_determine_partitions` to `pyarrow.py` [iceberg-python]

2024-07-11 Thread via GitHub
HonahX closed issue #896: Move `_determine_partitions` to `pyarrow.py` URL: https://github.com/apache/iceberg-python/issues/896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Rename `data_sequence_number` to `sequence_number` [iceberg-python]

2024-07-11 Thread via GitHub
HonahX closed issue #893: Rename `data_sequence_number` to `sequence_number` URL: https://github.com/apache/iceberg-python/issues/893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Namespace not exists after creating the namespace by REST [iceberg]

2024-07-11 Thread via GitHub
nastra closed issue #10656: Namespace not exists after creating the namespace by REST URL: https://github.com/apache/iceberg/issues/10656 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Namespace not exists after creating the namespace by REST [iceberg]

2024-07-11 Thread via GitHub
nastra commented on issue #10656: URL: https://github.com/apache/iceberg/issues/10656#issuecomment-210304 The tabular catalog doesn't support nested namespaces like `Namespace.of("public", "default")`. Given that this is a vendor-specific thing, I'll go ahead and close this issue. --

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1671665042 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1673566838 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1671717882 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [I] iceberg-aws-bundle jar includes org.slf4j.LoggerFactory [iceberg]

2024-07-11 Thread via GitHub
nastra commented on issue #10534: URL: https://github.com/apache/iceberg/issues/10534#issuecomment-268080 > Typically, I'd expect iceberg-aws-bundle to declare a dependency on slf4j-api The general idea of `iceberg-aws-bundle` is to be a fat jar that bundles everything related to

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1670390955 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [PR] Build: Declare avro as an api dependency of iceberg-core [iceberg]

2024-07-11 Thread via GitHub
nastra commented on PR #10573: URL: https://github.com/apache/iceberg/pull/10573#issuecomment-329253 Adding some additional context around `api` vs `implementation` for other reviewers: > Dependencies appearing in the api configurations will be transitively exposed to consumers o

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #902: URL: https://github.com/apache/iceberg-python/pull/902#discussion_r1673729169 ## tests/integration/test_deletes.py: ## @@ -291,7 +291,7 @@ def test_partitioned_table_positional_deletes_sequence_number(spark: SparkSessio assert snapshots[

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1673737827 ## pyiceberg/io/pyarrow.py: ## @@ -937,7 +937,7 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType: else: raise TypeError

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
findepi commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673751208 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + + Review Comment: Both m

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
findepi commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673753534 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: to leverage possessi

Re: [PR] DynConstructors cleanup [iceberg]

2024-07-11 Thread via GitHub
findepi commented on code in PR #10542: URL: https://github.com/apache/iceberg/pull/10542#discussion_r1673801761 ## common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java: ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

2024-07-11 Thread via GitHub
Fokko merged PR #902: URL: https://github.com/apache/iceberg-python/pull/902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Spec: Clarify which columns can be used for equality delete files. [iceberg]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1673839996 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify deleted rows

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1673841605 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2 table.

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1673855957 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2 table.

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1673855957 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2 table.

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-698145 @lurnagao-dahua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Support building with Java 21 [iceberg]

2024-07-11 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1673871100 ## baseline.gradle: ## @@ -46,7 +46,13 @@ subprojects { apply plugin: 'com.palantir.baseline-exact-dependencies' apply plugin: 'com.palantir.baseline-release-com

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673884503 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: Thanks, I see.

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673885080 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + + Review Comment:

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
nastra commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673890383 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: I think for this PR w

Re: [PR] Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1673893448 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2 flow

Re: [PR] Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1673893448 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2 flow

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673906437 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: @nastra, yep,

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673906437 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: @nastra, yep,

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673906437 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: @nastra, yep,

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673906437 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + Review Comment: @nastra, yep,

[PR] Update Snapshot Retention Properties [iceberg-python]

2024-07-11 Thread via GitHub
chinmay-bhat opened a new pull request, #913: URL: https://github.com/apache/iceberg-python/pull/913 WIP - will be ready to review once rollback PR 758 is merged into main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Update checkstyle definition [iceberg]

2024-07-11 Thread via GitHub
findepi commented on code in PR #10681: URL: https://github.com/apache/iceberg/pull/10681#discussion_r1673961429 ## .baseline/checkstyle/checkstyle.xml: ## @@ -480,23 +475,20 @@ - + -

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673885080 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + + Review Comment:

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10673: URL: https://github.com/apache/iceberg/pull/10673#discussion_r1673885080 ## .baseline/checkstyle/checkstyle.xml: ## @@ -284,6 +284,10 @@ + + + Review Comment:

[PR] Expire Snapshots [iceberg-python]

2024-07-11 Thread via GitHub
chinmay-bhat opened a new pull request, #914: URL: https://github.com/apache/iceberg-python/pull/914 WIP - [x] initial skeleton - [ ] clean expired snapshots - [ ] write tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Update checkstyle definition [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10681: URL: https://github.com/apache/iceberg/pull/10681#discussion_r1673979787 ## .baseline/checkstyle/checkstyle.xml: ## @@ -480,23 +475,20 @@ - + -

Re: [PR] Update checkstyle definition [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on code in PR #10681: URL: https://github.com/apache/iceberg/pull/10681#discussion_r1673979787 ## .baseline/checkstyle/checkstyle.xml: ## @@ -480,23 +475,20 @@ - + -

Re: [PR] Update checkstyle definition [iceberg]

2024-07-11 Thread via GitHub
findepi commented on code in PR #10681: URL: https://github.com/apache/iceberg/pull/10681#discussion_r1673988656 ## .baseline/checkstyle/checkstyle.xml: ## @@ -480,23 +475,20 @@ - + -

Re: [PR] Build: don't include slf4j-api in bundled JARs [iceberg]

2024-07-11 Thread via GitHub
bryanck commented on PR #10665: URL: https://github.com/apache/iceberg/pull/10665#issuecomment-975393 I think this makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Add checkstyle rule for uppercase constant fields [iceberg]

2024-07-11 Thread via GitHub
nastra merged PR #10673: URL: https://github.com/apache/iceberg/pull/10673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Expose table incremental scan for appends API in SerializableTable [iceberg]

2024-07-11 Thread via GitHub
nastra commented on PR #10682: URL: https://github.com/apache/iceberg/pull/10682#issuecomment-981956 @deniskuzZ can you add the missing `@Override` please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Build: don't include slf4j-api in bundled JARs [iceberg]

2024-07-11 Thread via GitHub
bryanck merged PR #10665: URL: https://github.com/apache/iceberg/pull/10665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] iceberg-aws-bundle jar includes org.slf4j.LoggerFactory [iceberg]

2024-07-11 Thread via GitHub
bryanck closed issue #10534: iceberg-aws-bundle jar includes org.slf4j.LoggerFactory URL: https://github.com/apache/iceberg/issues/10534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Fix NPE during conflict handling of NULL partitions [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10680: URL: https://github.com/apache/iceberg/pull/10680#discussion_r1674048472 ## core/src/main/java/org/apache/iceberg/util/PartitionSet.java: ## @@ -200,7 +200,8 @@ public String toString() { StringBuilder partitionStringBuilder =

Re: [I] iceberg-aws-bundle jar includes org.slf4j.LoggerFactory [iceberg]

2024-07-11 Thread via GitHub
bryanck closed issue #10534: iceberg-aws-bundle jar includes org.slf4j.LoggerFactory URL: https://github.com/apache/iceberg/issues/10534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Expose table incremental scan for appends API in SerializableTable [iceberg]

2024-07-11 Thread via GitHub
nastra commented on code in PR #10682: URL: https://github.com/apache/iceberg/pull/10682#discussion_r1674049469 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -278,6 +278,10 @@ public TableScan newScan() { return lazyTable().newScan(); } + publ

[PR] Spark timestamptz discrepancy [iceberg-python]

2024-07-11 Thread via GitHub
syun64 opened a new pull request, #915: URL: https://github.com/apache/iceberg-python/pull/915 I ran into this issue when I was trying to improve our tests for verifying write integrity for the timestamp types following our recent implementations for supporting inputs of other precisions an

Re: [PR] Bump mypy-boto3-glue from 1.34.136 to 1.34.143 [iceberg-python]

2024-07-11 Thread via GitHub
Fokko merged PR #912: URL: https://github.com/apache/iceberg-python/pull/912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[I] Detecting duplicates in the Flink Data Stream API [iceberg]

2024-07-11 Thread via GitHub
lkokhreidze opened a new issue, #10683: URL: https://github.com/apache/iceberg/issues/10683 ### Query engine Flink ### Question Hi, I was wondering if there's a way we could detect if ongoing batch written to the Iceberg table would perform the upsert? Context:

Re: [PR] Will not be merged: Spark timestamptz discrepancy [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on PR #915: URL: https://github.com/apache/iceberg-python/pull/915#issuecomment-2223035079 Here's the dataframe schema that represents the Iceberg tables that are loaded through: 1. Spark Iceberg 2. PyIceberg Spark Iceberg loads both timestamptz and timestamp ty

Re: [PR] Update checkstyle definition [iceberg]

2024-07-11 Thread via GitHub
attilakreiner commented on PR #10681: URL: https://github.com/apache/iceberg/pull/10681#issuecomment-2223052922 Merged in the `main` branch and fixed `ConstantName`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
snazy commented on PR #10603: URL: https://github.com/apache/iceberg/pull/10603#issuecomment-2223065233 I've updated the PR to mention "2.0". The CI failures look unrelated, but I don't have the power to rerun those. -- This is an automated message from the Apache Git Service. To re

Re: [PR] Core: Fix NPE during conflict handling of NULL partitions [iceberg]

2024-07-11 Thread via GitHub
boroknagyz commented on code in PR #10680: URL: https://github.com/apache/iceberg/pull/10680#discussion_r1674098559 ## core/src/main/java/org/apache/iceberg/util/PartitionSet.java: ## @@ -200,7 +200,8 @@ public String toString() { StringBuilder partitionStringBuilder

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-11 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1674134865 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -112,12 +112,6 @@ public void testSpecValues() { .as("Spec example: hash(2017-11-16T

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-11 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1674135249 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -165,6 +159,68 @@ public void testLong() { .isEqualTo(hashBytes(buffer.array()));

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-11 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1674135937 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -165,6 +159,68 @@ public void testLong() { .isEqualTo(hashBytes(buffer.array()));

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-11 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2223123048 > overall this LGTM once comments in `TestBucketing` have been addressed I've addressed these. Thanks! -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Check if schema is compatible in `add_files` API [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #907: URL: https://github.com/apache/iceberg-python/pull/907#discussion_r1674162322 ## pyiceberg/io/pyarrow.py: ## @@ -2026,6 +2072,8 @@ def parquet_files_to_data_files(io: FileIO, table_metadata: TableMetadata, file_ f"Cannot add

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-11 Thread via GitHub
danielcweeks commented on PR #10351: URL: https://github.com/apache/iceberg/pull/10351#issuecomment-2223231870 Thanks @bryanck and @fqaiser94. It's really great to get this one in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-11 Thread via GitHub
danielcweeks merged PR #10351: URL: https://github.com/apache/iceberg/pull/10351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
jackye1995 commented on PR #10603: URL: https://github.com/apache/iceberg/pull/10603#issuecomment-2223257678 Can you try rebase to see if it fixes the CI? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] API: add resultSchema() method to StructTransform [iceberg]

2024-07-11 Thread via GitHub
stevenzwu commented on PR #10496: URL: https://github.com/apache/iceberg/pull/10496#issuecomment-2223275495 hmm. the `TestSparkDataFile` failed after this change. ``` Caused by: org.apache.iceberg.exceptions.ValidationException: Invalid schema: multiple fields for name ts: 9 and 9

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
danielcweeks commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1674264095 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1674277456 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2 flow

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
danielcweeks commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1674279099 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,22 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2

Re: [PR] Core: use bulk delete when removing old metadata.json files [iceberg]

2024-07-11 Thread via GitHub
dramaticlly commented on PR #10679: URL: https://github.com/apache/iceberg/pull/10679#issuecomment-2223373221 > I think this is fine, I mentioned this another similar PR though so I thought I would note it here as well. We need to make sure our test case now runs with Filesystems that both

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674334862 ## pyiceberg/table/__init__.py: ## @@ -528,10 +528,6 @@ def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT) )

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on PR #910: URL: https://github.com/apache/iceberg-python/pull/910#issuecomment-2223408958 @Fokko @HonahX - thank you for your reviews. I've updated the integration test to make the [check more comprehensive](https://github.com/apache/iceberg-python/pull/910/files#diff-7f3d

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674351468 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.type

Re: [I] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on issue #10685: URL: https://github.com/apache/iceberg/issues/10685#issuecomment-2223424034 @Fokko @nastra I'm working on the gradle update PR. Do you mind guys to assign this ticket to me and set the milestone to 1.6.0 ? Thanks ! -- This is an automated message from t

Re: [PR] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on PR #10686: URL: https://github.com/apache/iceberg/pull/10686#issuecomment-2223435089 This PR closes #10686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
jbonofre commented on PR #10686: URL: https://github.com/apache/iceberg/pull/10686#issuecomment-2223435767 @Fokko @nastra thanks in advance gentlemen ! 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Running MERGE INTO with more than one WHEN condition fails if the number of columns in the target table is > 321 [iceberg]

2024-07-11 Thread via GitHub
andreaschiappacasse commented on issue #10294: URL: https://github.com/apache/iceberg/issues/10294#issuecomment-2223558105 @krishan711 we ended up using spark instead of athena to do the upsert/delete operation. It is still very unfortunate because it is much more expensive and adds some co

Re: [PR] Core: Expose table incremental scan for appends API in SerializableTable [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10682: URL: https://github.com/apache/iceberg/pull/10682#discussion_r1674500091 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -278,6 +278,10 @@ public TableScan newScan() { return lazyTable().newScan(); } + p

Re: [PR] Core: Expose table incremental scan for appends API in SerializableTable [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10682: URL: https://github.com/apache/iceberg/pull/10682#discussion_r1674506153 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -278,6 +278,10 @@ public TableScan newScan() { return lazyTable().newScan(); } + p

Re: [PR] Core: Expose table incremental scan for appends API in SerializableTable [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10682: URL: https://github.com/apache/iceberg/pull/10682#discussion_r1674506153 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -278,6 +278,10 @@ public TableScan newScan() { return lazyTable().newScan(); } + p

Re: [PR] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
Fokko merged PR #10686: URL: https://github.com/apache/iceberg/pull/10686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
Fokko closed issue #10685: Upgrade to Gradle 8.9 URL: https://github.com/apache/iceberg/issues/10685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: iss

Re: [PR] Upgrade to Gradle 8.9 [iceberg]

2024-07-11 Thread via GitHub
Fokko commented on PR #10686: URL: https://github.com/apache/iceberg/pull/10686#issuecomment-2223698571 Thanks @jbonofre for bumping Gradle here, and thanks @nastra, @ajantha-bhat and @snazy for the prompt review 🚀 -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] OpenAPI: Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-11 Thread via GitHub
snazy commented on PR #10603: URL: https://github.com/apache/iceberg/pull/10603#issuecomment-2223703737 > Can you try rebase to see if it fixes the CI? CI looking good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2223715282 LGTM +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1674554724 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -381,6 +386,46 @@ public void testCustomCatalog() throws IOException { testInputF

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1674558387 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -381,6 +386,56 @@ public void testCustomCatalog() throws IOException { testInputF

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1674565642 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -381,6 +386,56 @@ public void testCustomCatalog() throws IOException { testI

Re: [PR] mr:Fix issues 10639 [iceberg]

2024-07-11 Thread via GitHub
deniskuzZ commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1674568873 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -381,6 +386,56 @@ public void testCustomCatalog() throws IOException { testInputF

Re: [PR] Check if schema is compatible in `add_files` API [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #907: URL: https://github.com/apache/iceberg-python/pull/907#discussion_r1674603748 ## pyiceberg/io/pyarrow.py: ## @@ -2026,6 +2072,8 @@ def parquet_files_to_data_files(io: FileIO, table_metadata: TableMetadata, file_ f"Cannot add

Re: [PR] Check if schema is compatible in `add_files` API [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #907: URL: https://github.com/apache/iceberg-python/pull/907#discussion_r1674607135 ## pyiceberg/io/pyarrow.py: ## @@ -166,6 +166,7 @@ ONE_MEGABYTE = 1024 * 1024 BUFFER_SIZE = "buffer-size" + Review Comment: ```suggestion ``` -- This

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674682639 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.types

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674684198 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.types

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674686353 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.types

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674689843 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.type

Re: [PR] Support Spark Column Stats [iceberg]

2024-07-11 Thread via GitHub
singhpk234 commented on code in PR #10659: URL: https://github.com/apache/iceberg/pull/10659#discussion_r1674683253 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -175,7 +181,25 @@ public Statistics estimateStatistics() { protected Sta

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674691317 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.type

Re: [PR] Support Spark Column Stats [iceberg]

2024-07-11 Thread via GitHub
szehon-ho commented on code in PR #10659: URL: https://github.com/apache/iceberg/pull/10659#discussion_r1674693714 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -97,6 +117,36 @@ public static Object[][] parameters() { }; }

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
Fokko commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674710445 ## pyiceberg/io/pyarrow.py: ## @@ -1320,7 +1321,16 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array: and pa.types

Re: [I] Support writing to a branch [iceberg-python]

2024-07-11 Thread via GitHub
kevinjqliu commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2224059233 @vinjai yes! please go ahead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [Docs] Add examples for DataFrame branch writes [iceberg]

2024-07-11 Thread via GitHub
szehon-ho commented on code in PR #10644: URL: https://github.com/apache/iceberg/pull/10644#discussion_r1674784383 ## docs/docs/spark-writes.md: ## @@ -332,6 +332,30 @@ The writer must enable the `mergeSchema` option. ```scala data.writeTo("prod.db.sample").option("mergeSchema

Re: [PR] support PyArrow timestamptz with Etc/UTC [iceberg-python]

2024-07-11 Thread via GitHub
syun64 commented on code in PR #910: URL: https://github.com/apache/iceberg-python/pull/910#discussion_r1674797591 ## pyiceberg/io/pyarrow.py: ## @@ -1296,31 +1297,49 @@ def to_requested_schema( class ArrowProjectionVisitor(SchemaWithPartnerVisitor[pa.Array, Optional[pa.Ar

[PR] Bump coverage from 7.5.4 to 7.6.0 [iceberg-python]

2024-07-11 Thread via GitHub
dependabot[bot] opened a new pull request, #917: URL: https://github.com/apache/iceberg-python/pull/917 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.4 to 7.6.0. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

  1   2   >