Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1497038356 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +212,31 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [I] Merge into using the exactly dataset copy the entire data [iceberg]

2024-02-20 Thread via GitHub
fengguangyuan commented on issue #9736: URL: https://github.com/apache/iceberg/issues/9736#issuecomment-1956058615 >The tmp_data is literally the same data of warehouse.data, when running this code I would expected no changes in the dataset because it didn't match anything. However my parqu

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1497031984 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +212,31 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core: Add EnvironmentContext to commit summary [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9273: URL: https://github.com/apache/iceberg/pull/9273#discussion_r1497028069 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java: ## @@ -109,6 +110,7 @@ public void testRewriteDataFi

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1497024133 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + desc

Re: [PR] Flink: Incrementally rewrite data files in streaming. [iceberg]

2024-02-20 Thread via GitHub
lurnagao commented on PR #3323: URL: https://github.com/apache/iceberg/pull/3323#issuecomment-1956030230 > > Hello, may I ask if the rewrite is asynchronous or synchronous > > The rewrite is performed asynchronously. @lurnagao Hi! I'm a bit confused. Is the visible time of the

[PR] Add workflow for cargo audit [iceberg-rust]

2024-02-20 Thread via GitHub
sdd opened a new pull request, #217: URL: https://github.com/apache/iceberg-rust/pull/217 Resolves https://github.com/apache/iceberg-rust/issues/209. Workflow file copied directly from https://github.com/apache/arrow-datafusion/blob/main/.github/workflows/audit.yml -- This is an au

Re: [PR] Core: Fix EncryptingFileIO factory method. [iceberg]

2024-02-20 Thread via GitHub
nastra merged PR #9757: URL: https://github.com/apache/iceberg/pull/9757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] docs: Add basic README for all crates [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #215: URL: https://github.com/apache/iceberg-rust/pull/215#issuecomment-1956011515 cc @Fokko, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] docs: Add basic README for all crates [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on code in PR #215: URL: https://github.com/apache/iceberg-rust/pull/215#discussion_r1496997143 ## crates/iceberg/README.md: ## @@ -0,0 +1,31 @@ + + +# Apache Iceberg Official Native Rust Implementation + +[![crates.io](https://img.shields.io/crates/v/iceberg.sv

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
ajantha-bhat commented on PR #9765: URL: https://github.com/apache/iceberg/pull/9765#issuecomment-1956009664 Keeping to V0 as default seems to be the right choice 👍 . I think only thing missing is a testcase with `initializeCatalogTables = false` as Eduard requested. We can para

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

2024-02-20 Thread via GitHub
ggershinsky commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1496995688 ## api/src/main/java/org/apache/iceberg/Files.java: ## @@ -95,26 +95,40 @@ public String toString() { } public static InputFile localInput(File file) { -

Re: [I] Add a download page [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on issue #213: URL: https://github.com/apache/iceberg-rust/issues/213#issuecomment-1955998721 The ASF Infra has their requirement for this page, we can take reference from opendal: https://github.com/apache/opendal/blob/main/website/src/pages/download.md -- This is an au

Re: [PR] docs: Upload crates [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #211: URL: https://github.com/apache/iceberg-rust/pull/211#issuecomment-1955997268 > Is it possible to include other stuff like uploading tars in svn? This task involves the password for PMC's ASF account. It's best to avoid it. -- This is an automated message

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 closed issue #180: Tracking issues of Iceberg Rust 0.2.0 Release URL: https://github.com/apache/iceberg-rust/issues/180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1955978672 Close this issue since all steps have finished, thanks everyone's effort! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] docs: Upload crates [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on PR #211: URL: https://github.com/apache/iceberg-rust/pull/211#issuecomment-1955967003 > Thanks! By the way, we can use CI to test and publish the crates while pushing tag like https://github.com/apache/opendal/blob/main/.github/workflows/release_rust.yml. >

Re: [I] Add a download page [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on issue #213: URL: https://github.com/apache/iceberg-rust/issues/213#issuecomment-1955962755 I'll fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1955958747 > What we do at Python/Java, is create a branch for each major/minor version. If there are any bugs then we can backport them to the branch, and do a quick release. This way you

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1955956400 > > Upgrading minor version means add new features without breaking change, e.g. 0.2.0 -> 0.3.0. > > This is a breaking change in semver definition. I took a look at

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496932026 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496930464 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496928516 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496928516 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496926708 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496925205 ## tests/integration/test_partitioning_key.py: ## @@ -0,0 +1,722 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 commented on code in PR #453: URL: https://github.com/apache/iceberg-python/pull/453#discussion_r1496923619 ## pyiceberg/partitioning.py: ## @@ -215,3 +240,53 @@ def assign_fresh_partition_spec_ids(spec: PartitionSpec, old_schema: Schema, fre ) )

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on PR #9765: URL: https://github.com/apache/iceberg/pull/9765#issuecomment-1955906963 @rdblue @danielcweeks @nastra @ajantha-bhat I did a first update of the PR according to Ryan's comments. I'm still working on a new test (the PR will be updated soon with this test). -

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-20 Thread via GitHub
zinking commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1496914794 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -507,4 +645,54 @@ public int totalGroupCount() { retur

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-20 Thread via GitHub
zinking commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1496914794 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -507,4 +645,54 @@ public int totalGroupCount() { retur

Re: [PR] Flink 1.18: Fix iceberg source plan parallelism not effective. [iceberg]

2024-02-20 Thread via GitHub
stevenzwu commented on PR #9761: URL: https://github.com/apache/iceberg/pull/9761#issuecomment-1955884328 thanks @Reo-LEI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Flink 1.18: Fix iceberg source plan parallelism not effective. [iceberg]

2024-02-20 Thread via GitHub
stevenzwu merged PR #9761: URL: https://github.com/apache/iceberg/pull/9761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Flink 1.18: Fix continuous enumerator lost enumeration history state when restore from checkpoint. [iceberg]

2024-02-20 Thread via GitHub
stevenzwu commented on PR #9762: URL: https://github.com/apache/iceberg/pull/9762#issuecomment-1955875029 thanks @Reo-LEI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Flink 1.18: Fix continuous enumerator lost enumeration history state when restore from checkpoint. [iceberg]

2024-02-20 Thread via GitHub
stevenzwu merged PR #9762: URL: https://github.com/apache/iceberg/pull/9762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
stevenzwu commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496881682 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
stevenzwu commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496878897 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
stevenzwu commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496874140 ## open-api/rest-catalog-open-api.py: ## @@ -905,6 +973,28 @@ class CreateTableRequest(BaseModel): properties: Optional[Dict[str, str]] = None +class PlanCon

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-20 Thread via GitHub
anupam-saini commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1496863736 ## pyiceberg/schema.py: ## @@ -1213,34 +1213,50 @@ def build_position_accessors(schema_or_type: Union[Schema, IcebergType]) -> Dict return visit(schema_

Re: [I] Is it possible to add a set of existing partitioned parquet files to the Iceberg table via the Java Standalone API [iceberg]

2024-02-20 Thread via GitHub
manuzhang commented on issue #9763: URL: https://github.com/apache/iceberg/issues/9763#issuecomment-1955830733 You may want to check out [AddFilesProcedure#importFileTable](https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/AddFiles

Re: [PR] Spark 3.5: Set log level to WARN for rewrite task failure with partial progress [iceberg]

2024-02-20 Thread via GitHub
manuzhang closed pull request #9400: Spark 3.5: Set log level to WARN for rewrite task failure with partial progress URL: https://github.com/apache/iceberg/pull/9400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.5: Add max allowed failed commits to RewriteDataFiles when partial progress is enabled [iceberg]

2024-02-20 Thread via GitHub
manuzhang commented on code in PR #9611: URL: https://github.com/apache/iceberg/pull/9611#discussion_r1496849646 ## api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java: ## @@ -52,6 +52,13 @@ public interface RewriteDataFiles int PARTIAL_PROGRESS_MAX_COMMITS_D

[I] Create table error when using HDFS [iceberg-python]

2024-02-20 Thread via GitHub
wuxiaocheng0506 opened a new issue, #459: URL: https://github.com/apache/iceberg-python/issues/459 ### Apache Iceberg version None ### Please describe the bug 🐞 version: 0.6.0 I using hive metastore and HDFS file IO. When I create table, an error occur: >

Re: [PR] Build: Bump getdaft from 0.2.15 to 0.2.16 [iceberg-python]

2024-02-20 Thread via GitHub
HonahX merged PR #456: URL: https://github.com/apache/iceberg-python/pull/456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Bump pre-commit and such [iceberg-python]

2024-02-20 Thread via GitHub
HonahX commented on PR #442: URL: https://github.com/apache/iceberg-python/pull/442#issuecomment-1955711221 Do we want to merge this first or wait for the Ruff issue to be solved? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Build: Bump getdaft from 0.2.15 to 0.2.16 [iceberg-python]

2024-02-20 Thread via GitHub
HonahX commented on PR #456: URL: https://github.com/apache/iceberg-python/pull/456#issuecomment-1955690116 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Build: Bump coverage from 7.4.1 to 7.4.2 [iceberg-python]

2024-02-20 Thread via GitHub
HonahX merged PR #457: URL: https://github.com/apache/iceberg-python/pull/457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Accept pyarrow LargeListType and FixedSizeListType [iceberg-python]

2024-02-20 Thread via GitHub
HonahX commented on code in PR #458: URL: https://github.com/apache/iceberg-python/pull/458#discussion_r1496752910 ## pyiceberg/io/pyarrow.py: ## @@ -689,15 +689,28 @@ def _(obj: pa.StructType, visitor: PyArrowSchemaVisitor[T]) -> T: return visitor.struct(obj, results)

Re: [I] Support Parquet modular encryption [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #1413: URL: https://github.com/apache/iceberg/issues/1413#issuecomment-1955455734 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Clean orphan data files [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #873: URL: https://github.com/apache/iceberg/issues/873#issuecomment-1955455131 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] [InputFormat Followup] Add residual evaluation for Iceberg generics [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] closed issue #866: [InputFormat Followup] Add residual evaluation for Iceberg generics URL: https://github.com/apache/iceberg/issues/866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [Proposal] An iceberg-unstructured module [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #859: URL: https://github.com/apache/iceberg/issues/859#issuecomment-1955455001 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Flink: Improvements for iceberg sink connector. [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #1403: URL: https://github.com/apache/iceberg/issues/1403#issuecomment-1955455614 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Consider a builder in TableMetadata [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #1412: URL: https://github.com/apache/iceberg/issues/1412#issuecomment-1955455673 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [InputFormat Followup] Add residual evaluation for Iceberg generics [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #866: URL: https://github.com/apache/iceberg/issues/866#issuecomment-1955455066 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Clean orphan data files [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] closed issue #873: Clean orphan data files URL: https://github.com/apache/iceberg/issues/873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] [Proposal] An iceberg-unstructured module [iceberg]

2024-02-20 Thread via GitHub
github-actions[bot] closed issue #859: [Proposal] An iceberg-unstructured module URL: https://github.com/apache/iceberg/issues/859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496677678 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,280 @@ components: type: integer format: int64 +BooleanTypeValue: + type: boo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496670351 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,280 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] Add support for List as a data type in the writer [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on issue #451: URL: https://github.com/apache/iceberg-python/issues/451#issuecomment-1955288007 The list data type is supported in PyIceberg, but there is a bug in the mapping method, I believe https://github.com/apache/iceberg-python/pull/458 will fix the issue. -

Re: [PR] Accept pyarrow LargeListType and FixedSizeListType [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on code in PR #458: URL: https://github.com/apache/iceberg-python/pull/458#discussion_r149049 ## pyiceberg/io/pyarrow.py: ## @@ -689,15 +689,28 @@ def _(obj: pa.StructType, visitor: PyArrowSchemaVisitor[T]) -> T: return visitor.struct(obj, resul

[PR] Accept pyarrow LargeListType and FixedSizeListType [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala opened a new pull request, #458: URL: https://github.com/apache/iceberg-python/pull/458 closes: #451 The C++ implementation of Arrow has different types of lists (https://arrow.apache.org/docs/cpp/api/datatype.html#_CPPv4N5arrow12BaseListTypeE), but these types are not

[PR] Build: Bump coverage from 7.4.1 to 7.4.2 [iceberg-python]

2024-02-20 Thread via GitHub
dependabot[bot] opened a new pull request, #457: URL: https://github.com/apache/iceberg-python/pull/457 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.4.1 to 7.4.2. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

[PR] Build: Bump getdaft from 0.2.15 to 0.2.16 [iceberg-python]

2024-02-20 Thread via GitHub
dependabot[bot] opened a new pull request, #456: URL: https://github.com/apache/iceberg-python/pull/456 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.15 to 0.2.16. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-02-20 Thread via GitHub
JGynther commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-1955236592 I found the same issue. However using Access Point S3 alias it seems to work just fine. Here is a minimal PyArrow example that reads a file: ```python from pyarrow imp

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-1955233783 I think the problem is in Pyarrow S3FileSystem, I found this open issue https://issues.apache.org/jira/browse/ARROW-9669 (moved to https://github.com/apache/arrow/issues/2

Re: [I] com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.nio.ByteBuffer [iceberg]

2024-02-20 Thread via GitHub
adevore3 commented on issue #9738: URL: https://github.com/apache/iceberg/issues/9738#issuecomment-1955227278 Sorry for the complex code sample. I verified that I can reproduce the error w/ a more simple datasource: ``` case class JobArchiveOsirisDatasource(minTs: Long, inputPartitions

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-02-20 Thread via GitHub
JGynther commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-1955209888 Here is my minimal code example (that fails): ```python from pyiceberg.table import StaticTable # Latest metadata file object = "iceberg/metadata/00068-b5e701c2-

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-1955204486 Could you share the full exception you get when you run the scan query? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] detect breaking changes [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on PR #394: URL: https://github.com/apache/iceberg-python/pull/394#issuecomment-1955199604 @syun64 I was on a merging spree, can you rebase once more? 😓 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump pytest from 7.4.4 to 8.0.0 [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on PR #393: URL: https://github.com/apache/iceberg-python/pull/393#issuecomment-1955196901 Yes, we can wait two weeks, and if there is no solution to this issue, we can discuss merging this PR or adopting a different solution. -- This is an automated message from t

Re: [PR] Feat: Implement `create_table_if_not_exists` [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #415: URL: https://github.com/apache/iceberg-python/pull/415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] check if table exist [iceberg-python]

2024-02-20 Thread via GitHub
Fokko closed issue #406: check if table exist URL: https://github.com/apache/iceberg-python/issues/406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: i

Re: [PR] Feat: Implement `create_table_if_not_exists` [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on code in PR #415: URL: https://github.com/apache/iceberg-python/pull/415#discussion_r1496586814 ## tests/catalog/test_rest.py: ## @@ -560,6 +560,64 @@ def test_create_table_409(rest_mock: Mocker, table_schema_simple: Schema) -> Non assert "Table already e

Re: [PR] Feat: Implement `create_table_if_not_exists` [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on PR #415: URL: https://github.com/apache/iceberg-python/pull/415#issuecomment-1955191588 I think it's ready 🤞 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on PR #453: URL: https://github.com/apache/iceberg-python/pull/453#issuecomment-1955191403 @jqin61 Nice! Thanks for working on this. It is getting late here, but this is on my list for tomorrow 👍 -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496583084 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496582673 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

[I] Implement incremental updating of table metadata [iceberg-python]

2024-02-20 Thread via GitHub
Fokko opened a new issue, #454: URL: https://github.com/apache/iceberg-python/issues/454 ### Feature Request / Improvement When we accumulate changes when doing updates to the metadata, we want to accumulate the changes in the metadata. For example, if you first add a new colum

[PR] PartitionKey [iceberg-python]

2024-02-20 Thread via GitHub
jqin61 opened a new pull request, #453: URL: https://github.com/apache/iceberg-python/pull/453 **Scope** Add PartitionKey class which: 1. is used to hold the raw partition field and values. This is for partitioned write. 2. converts the python values into iceberg-typed values. 3.

Re: [PR] Update NameMapping on update_schema() [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #441: URL: https://github.com/apache/iceberg-python/pull/441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Update NameMapping on update_schema() [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on code in PR #441: URL: https://github.com/apache/iceberg-python/pull/441#discussion_r1496576875 ## tests/table/test_name_mapping.py: ## @@ -238,3 +245,67 @@ def test_mapping_lookup_by_name(table_name_mapping_nested: NameMapping) -> None: with pytest.rai

Re: [PR] Reuse commit-uuid as the write-uuid [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #437: URL: https://github.com/apache/iceberg-python/pull/437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Change Append/Overwrite API to accept snapshot properties [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on code in PR #419: URL: https://github.com/apache/iceberg-python/pull/419#discussion_r1496570473 ## pyiceberg/table/__init__.py: ## @@ -1019,21 +1020,27 @@ def append(self, df: pa.Table) -> None: if len(self.spec().fields) > 0: raise ValueE

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496301006 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496301006 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

Re: [PR] Change Append/Overwrite API to accept snapshot properties [iceberg-python]

2024-02-20 Thread via GitHub
Gowthami03B commented on PR #419: URL: https://github.com/apache/iceberg-python/pull/419#issuecomment-1955150881 > @Gowthami03B Now the release is out, we can merge this. Could you fix the merge conflicts? Thanks! @Fokko Done! -- This is an automated message from the Apache Git Ser

[I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-02-20 Thread via GitHub
JGynther opened a new issue, #452: URL: https://github.com/apache/iceberg-python/issues/452 ### Feature Request / Improvement [S3 Access Points](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html) are a way to scale data access by providing additional endpoints

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496522604 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] Rest Catalog Refresh Access Token with stored CLIENT_ID and CLIENT_SECRET [iceberg-python]

2024-02-20 Thread via GitHub
Fokko closed issue #234: Rest Catalog Refresh Access Token with stored CLIENT_ID and CLIENT_SECRET URL: https://github.com/apache/iceberg-python/issues/234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Retry with new Access Token on 419 response [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #340: URL: https://github.com/apache/iceberg-python/pull/340 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Change Append/Overwrite API to accept snapshot properties [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on PR #419: URL: https://github.com/apache/iceberg-python/pull/419#issuecomment-1955098638 @Gowthami03B Now the release is out, we can merge this. Could you fix the merge conflicts? Thanks! -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Send X-Iceberg-Access-Delegation header to signal support for vended credentials/remote signing [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #436: URL: https://github.com/apache/iceberg-python/pull/436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump pytest from 7.4.4 to 8.0.0 [iceberg-python]

2024-02-20 Thread via GitHub
Fokko commented on PR #393: URL: https://github.com/apache/iceberg-python/pull/393#issuecomment-1955093871 Circling back to this now the release is out. I do like the lazy-fixture, and it solved some real problems. Also, the request seems to fix the problem, but erases all typing and is mor

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496493629 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Feat: Add fail_if_exists param to create_table [iceberg-python]

2024-02-20 Thread via GitHub
hussein-awala commented on PR #415: URL: https://github.com/apache/iceberg-python/pull/415#issuecomment-1955078993 I will update the PR then 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1496482611 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495091041 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1496474476 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496470078 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type: boo

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496459073 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

  1   2   3   >