Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-27 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1913476880 @RussellSpitzer Hello. can you check this? Tks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump org.testcontainers:testcontainers from 1.19.3 to 1.19.4 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9577: URL: https://github.com/apache/iceberg/pull/9577 Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.19.3 to 1.19.4. Release notes Sourced from https://github.com/testcontainers/t

[PR] Build: Bump org.assertj:assertj-core from 3.25.1 to 3.25.2 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9576: URL: https://github.com/apache/iceberg/pull/9576 Bumps [org.assertj:assertj-core](https://github.com/assertj/assertj) from 3.25.1 to 3.25.2. Release notes Sourced from https://github.com/assertj/assertj/releases";>org.assertj:assert

[PR] Build: Bump software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin from 1.0.1 to 2.0.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9575: URL: https://github.com/apache/iceberg/pull/9575 Bumps [software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin](https://github.com/aws/aws-s3-accessgrants-plugin-java-v2) from 1.0.1 to 2.0.0. Commits See full diff in h

[PR] Build: Bump arrow from 14.0.2 to 15.0.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9574: URL: https://github.com/apache/iceberg/pull/9574 Bumps `arrow` from 14.0.2 to 15.0.0. Updates `org.apache.arrow:arrow-memory-netty` from 14.0.2 to 15.0.0 Updates `org.apache.arrow:arrow-vector` from 14.0.2 to 15.0.0 Commits

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.23.2 to 2.23.7 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] closed pull request #9539: Build: Bump software.amazon.awssdk:bom from 2.23.2 to 2.23.7 URL: https://github.com/apache/iceberg/pull/9539 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.23.2 to 2.23.7 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] commented on PR #9539: URL: https://github.com/apache/iceberg/pull/9539#issuecomment-1913448213 Superseded by #9573. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump software.amazon.awssdk:bom from 2.23.2 to 2.23.12 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9573: URL: https://github.com/apache/iceberg/pull/9573 Bumps software.amazon.awssdk:bom from 2.23.2 to 2.23.12. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwar

Re: [PR] Build: Bump com.azure:azure-sdk-bom from 1.2.18 to 1.2.19 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] closed pull request #9261: Build: Bump com.azure:azure-sdk-bom from 1.2.18 to 1.2.19 URL: https://github.com/apache/iceberg/pull/9261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump com.azure:azure-sdk-bom from 1.2.18 to 1.2.19 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] commented on PR #9261: URL: https://github.com/apache/iceberg/pull/9261#issuecomment-1913448022 Superseded by #9571. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.2.3 to 5.3.1 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9572: URL: https://github.com/apache/iceberg/pull/9572 Bumps [org.apache.httpcomponents.client5:httpclient5](https://github.com/apache/httpcomponents-client) from 5.2.3 to 5.3.1. Changelog Sourced from https://github.com/apache/httpcompo

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.18 to 1.2.20 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9571: URL: https://github.com/apache/iceberg/pull/9571 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.18 to 1.2.20. Commits https://github.com/Azure/azure-sdk-for-java/commit/c93df912375fbe795589

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.4 to 3.14.5 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9570: URL: https://github.com/apache/iceberg/pull/9570 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.14.4 to 3.14.5. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/re

[PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.25.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9569: URL: https://github.com/apache/iceberg/pull/9569 Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 6.13.0 to 6.25.0. Commits https://github.com/diffplug/spotless/commit/cac8d8f8f2d2

[PR] Build: Bump nessie from 0.76.3 to 0.76.6 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9568: URL: https://github.com/apache/iceberg/pull/9568 Bumps `nessie` from 0.76.3 to 0.76.6. Updates `org.projectnessie.nessie:nessie-client` from 0.76.3 to 0.76.6 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.76.3 t

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.35.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] commented on PR #9468: URL: https://github.com/apache/iceberg/pull/9468#issuecomment-1913447218 Superseded by #9567. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.35.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] closed pull request #9468: Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.35.0 URL: https://github.com/apache/iceberg/pull/9468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.36.0 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9567: URL: https://github.com/apache/iceberg/pull/9567 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 4.42.0 to 5.36.0. Release notes Sourced from https://github.com/palantir/gradle-b

Re: [PR] Build: Bump mkdocs-material from 9.5.3 to 9.5.4 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] closed pull request #9533: Build: Bump mkdocs-material from 9.5.3 to 9.5.4 URL: https://github.com/apache/iceberg/pull/9533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Build: Bump mkdocs-material from 9.5.3 to 9.5.4 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] commented on PR #9533: URL: https://github.com/apache/iceberg/pull/9533#issuecomment-1913445848 Superseded by #9566. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump mkdocs-material from 9.5.3 to 9.5.5 [iceberg]

2024-01-27 Thread via GitHub
dependabot[bot] opened a new pull request, #9566: URL: https://github.com/apache/iceberg/pull/9566 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.3 to 9.5.5. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-ma

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-27 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1468709394 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-27 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1468708917 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-27 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1468708459 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-27 Thread via GitHub
fqaiser94 commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1468699416 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-27 Thread via GitHub
fqaiser94 commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1464225023 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software F

Re: [I] Suggestion for newbie getting started guide [iceberg]

2024-01-27 Thread via GitHub
github-actions[bot] commented on issue #761: URL: https://github.com/apache/iceberg/issues/761#issuecomment-1913380102 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [PR] Spec: Clarify which columns can be used for equality delete files. [iceberg]

2024-01-27 Thread via GitHub
emkornfield commented on PR #8981: URL: https://github.com/apache/iceberg/pull/8981#issuecomment-1913366693 @Fokko @rdblue would you mind reviewing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2024-01-27 Thread via GitHub
emkornfield commented on PR #8982: URL: https://github.com/apache/iceberg/pull/8982#issuecomment-1913366641 @Fokko @aokolnychyi would you mind taking a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-27 Thread via GitHub
emkornfield commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1468680676 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`

Re: [PR] Spec: add multi-arg transform support [iceberg]

2024-01-27 Thread via GitHub
emkornfield commented on code in PR #8579: URL: https://github.com/apache/iceberg/pull/8579#discussion_r1468680676 ## format/spec.md: ## @@ -1128,12 +1128,17 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`

[PR] Check the types when writing [iceberg-python]

2024-01-27 Thread via GitHub
Fokko opened a new pull request, #313: URL: https://github.com/apache/iceberg-python/pull/313 Annotations are not checked at runtime -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468632826 ## pyiceberg/catalog/__init__.py: ## @@ -512,6 +516,22 @@ def _check_for_overlap(removals: Optional[Set[str]], updates: Properties) -> Non if overlap:

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468632375 ## pyiceberg/catalog/__init__.py: ## @@ -512,6 +516,22 @@ def _check_for_overlap(removals: Optional[Set[str]], updates: Properties) -> Non if over

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468631158 ## pyiceberg/catalog/__init__.py: ## @@ -512,6 +516,22 @@ def _check_for_overlap(removals: Optional[Set[str]], updates: Properties) -> Non if over

Re: [I] Consolidate FileIO [iceberg-python]

2024-01-27 Thread via GitHub
Fokko commented on issue #310: URL: https://github.com/apache/iceberg-python/issues/310#issuecomment-1913318831 What would be your proposal? The [FileIO is an abstraction](https://tabular.io/blog/iceberg-fileio-cloud-native-tables/) layer to use different implementations for your needs. For

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468603355 ## pyiceberg/catalog/__init__.py: ## @@ -512,6 +516,22 @@ def _check_for_overlap(removals: Optional[Set[str]], updates: Properties) -> Non if overlap:

Re: [PR] Refactor to write APIs to default to `main` branch [iceberg-python]

2024-01-27 Thread via GitHub
Fokko commented on code in PR #312: URL: https://github.com/apache/iceberg-python/pull/312#discussion_r1468590647 ## pyiceberg/table/__init__.py: ## @@ -2279,12 +2280,14 @@ class _MergingSnapshotProducer: _parent_snapshot_id: Optional[int] _added_data_files: List[DataF

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468589185 ## tests/catalog/test_base.py: ## @@ -255,6 +258,10 @@ def catalog() -> InMemoryCatalog: return InMemoryCatalog("test.in.memory.catalog", **{"test.key": "test

Re: [I] Support writing to a branch [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913306420 also note `dev/provision.py` which is used for integration tests already have statements to create tags and branchs https://github.com/apache/iceberg-python/blob/9e039

Re: [I] Support writing to a branch [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913305829 We first need a `create branch` API. Then update places currently gated by `MAIN_BRANCH`. 1. https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f7

Re: [I] Support writing to a branch [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913304752 First pass, just refactoring #312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Refactor to write APIs to default to `main` branch [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu opened a new pull request, #312: URL: https://github.com/apache/iceberg-python/pull/312 Issue #306 (Support writing to a branch) First pass. This PR * Update string literals (`"main"`/`"branch"`/`"tag"`) to its corresponding constant * Default `append` and `overw

Re: [I] Support writing to a branch [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913303260 In order to write to a branch, the branch needs to be created first. From https://iceberg.apache.org/docs/latest/spark-writes/#writing-to-branches: > the branch

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468585851 ## pyiceberg/catalog/__init__.py: ## @@ -512,6 +516,22 @@ def _check_for_overlap(removals: Optional[Set[str]], updates: Properties) -> Non if overlap:

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468585068 ## tests/io/test_pyarrow_visitor.py: ## @@ -572,3 +477,15 @@ def test_pyarrow_schema_to_schema_missing_ids_using_name_mapping_nested_missing_ with pytest.rais

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468546304 ## mkdocs/docs/api.md: ## @@ -146,6 +146,26 @@ catalog.create_table( ) ``` +One can also create an Iceberg table using a pyarrow schema: Review Comment:

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-27 Thread via GitHub
kevinjqliu commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1468547838 ## tests/io/test_pyarrow_visitor.py: ## @@ -572,3 +477,15 @@ def test_pyarrow_schema_to_schema_missing_ids_using_name_mapping_nested_missing_ with pytest.

Re: [I] iceberg-flink: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-01-27 Thread via GitHub
ilyasahsan123 commented on issue #9087: URL: https://github.com/apache/iceberg/issues/9087#issuecomment-1913138055 Hi @nastra, I've submitted a pull request [PR #9565](https://github.com/apache/iceberg/pull/9565) to migrate unittests to JUnit5 in Flink v1.16. I'm considering s

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-01-27 Thread via GitHub
JanKaul commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1913116304 Hi @szehon-ho, thanks for trying to move the process of reaching consensus along. To be honest, I don't know how the community normally reaches consensus on these kinds of topics. Bu

Re: [I] Slowness when loading table from S3 [iceberg-python]

2024-01-27 Thread via GitHub
itaise commented on issue #220: URL: https://github.com/apache/iceberg-python/issues/220#issuecomment-1913088810 Hi, Actually i didnt. Will love to hear if there is a way. We would like to get only table metadata in the fastest way possible (for user facing UIs) בתאריך יום ו׳, 26

Re: [PR] Build: Bump coverage from 7.4.0 to 7.4.1 [iceberg-python]

2024-01-27 Thread via GitHub
Fokko merged PR #307: URL: https://github.com/apache/iceberg-python/pull/307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump boto3 from 1.34.22 to 1.34.27 [iceberg-python]

2024-01-27 Thread via GitHub
Fokko merged PR #308: URL: https://github.com/apache/iceberg-python/pull/308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump boto3 from 1.34.22 to 1.34.27 [iceberg-python]

2024-01-27 Thread via GitHub
Fokko commented on PR #308: URL: https://github.com/apache/iceberg-python/pull/308#issuecomment-1913069104 Probably we have to update `aiobotocore` as well: https://github.com/aio-libs/aiobotocore/blob/master/setup.py#L10 -- This is an automated message from the Apache Git Service. To res

Re: [PR] Build: Bump pypa/cibuildwheel from 2.16.2 to 2.16.3 [iceberg-python]

2024-01-27 Thread via GitHub
Fokko merged PR #309: URL: https://github.com/apache/iceberg-python/pull/309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.