Re: [I] gc.enabled property is set to false by default for Apache Iceberg table created in Nessie Catalog [iceberg]

2024-04-13 Thread via GitHub
clintf1982 commented on issue #9562: URL: https://github.com/apache/iceberg/issues/9562#issuecomment-2053922595 @ajantha-bhat Thanks a lot:) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.5 to 3.15.1 [iceberg]

2024-04-13 Thread via GitHub
Fokko commented on PR #10095: URL: https://github.com/apache/iceberg/pull/10095#issuecomment-2053908709 Moving this forward, thanks @sfc-gh-dhuo for letting us know 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.5 to 3.15.1 [iceberg]

2024-04-13 Thread via GitHub
Fokko merged PR #10095: URL: https://github.com/apache/iceberg/pull/10095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1564467550 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received any

[PR] Build: Bump org.springframework:spring-web from 5.3.33 to 5.3.34 [iceberg]

2024-04-13 Thread via GitHub
dependabot[bot] opened a new pull request, #10139: URL: https://github.com/apache/iceberg/pull/10139 Bumps [org.springframework:spring-web](https://github.com/spring-projects/spring-framework) from 5.3.33 to 5.3.34. Release notes Sourced from https://github.com/spring-projects/spr

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.26 [iceberg]

2024-04-13 Thread via GitHub
dependabot[bot] commented on PR #10093: URL: https://github.com/apache/iceberg/pull/10093#issuecomment-2053905136 Superseded by #10138. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.26 [iceberg]

2024-04-13 Thread via GitHub
dependabot[bot] closed pull request #10093: Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.26 URL: https://github.com/apache/iceberg/pull/10093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.31 [iceberg]

2024-04-13 Thread via GitHub
dependabot[bot] opened a new pull request, #10138: URL: https://github.com/apache/iceberg/pull/10138 Bumps software.amazon.awssdk:bom from 2.25.21 to 2.25.31. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1564461947 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received any

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1564459557 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received any a

Re: [PR] [0.6.x] Backport #585 [iceberg-python]

2024-04-13 Thread via GitHub
Fokko merged PR #586: URL: https://github.com/apache/iceberg-python/pull/586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] [0.6.x] Backport #529 and #597 [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on PR #605: URL: https://github.com/apache/iceberg-python/pull/605#issuecomment-2053902927 Thanks @HonahX for working on this. I think this is all we need to resume the 0.6.1 release -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [0.6.x] Backport #529 and #597 [iceberg-python]

2024-04-13 Thread via GitHub
Fokko merged PR #605: URL: https://github.com/apache/iceberg-python/pull/605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Flink, Spark: Replace Boolean.getBoolean() with Boolean.parseBoolean() [iceberg]

2024-04-13 Thread via GitHub
amogh-jahagirdar merged PR #10136: URL: https://github.com/apache/iceberg/pull/10136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] expireSnapshots excepts with V2 when delete-after-commit.enabled=true and previous-versions-max=1 [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2493: URL: https://github.com/apache/iceberg/issues/2493#issuecomment-2053813762 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Merge into is Error。Error:java.lang.NoSuchMethodError [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2344: URL: https://github.com/apache/iceberg/issues/2344#issuecomment-2053813690 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Test: Add row-level API for TestTable to make the unit test more easy. [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2334: URL: https://github.com/apache/iceberg/issues/2334#issuecomment-2053813688 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Merge into is Error。Error:java.lang.NoSuchMethodError [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2344: Merge into is Error。Error:java.lang.NoSuchMethodError URL: https://github.com/apache/iceberg/issues/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] how to set Catalogs in structured streaming local model [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2322: URL: https://github.com/apache/iceberg/issues/2322#issuecomment-2053813669 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: Provide the fully covered SQL integration test cases to verify the data correctness. [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2313: Flink: Provide the fully covered SQL integration test cases to verify the data correctness. URL: https://github.com/apache/iceberg/issues/2313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] Flink: Support nested projection in iceberg table source [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2312: URL: https://github.com/apache/iceberg/issues/2312#issuecomment-2053813639 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Reading from iceberg table through spark thrift server using jdbc taking larger time [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2302: URL: https://github.com/apache/iceberg/issues/2302#issuecomment-2053813630 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Reading from iceberg table through spark thrift server using jdbc taking larger time [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2302: Reading from iceberg table through spark thrift server using jdbc taking larger time URL: https://github.com/apache/iceberg/issues/2302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Error while evolving partition column of a table [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2327: Error while evolving partition column of a table URL: https://github.com/apache/iceberg/issues/2327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Error while evolving partition column of a table [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #2327: URL: https://github.com/apache/iceberg/issues/2327#issuecomment-2053813681 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Test: Add row-level API for TestTable to make the unit test more easy. [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2334: Test: Add row-level API for TestTable to make the unit test more easy. URL: https://github.com/apache/iceberg/issues/2334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Flink: Support nested projection in iceberg table source [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #2312: Flink: Support nested projection in iceberg table source URL: https://github.com/apache/iceberg/issues/2312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Spark writes the Iceberg dual partition table to report an error [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] commented on issue #1894: URL: https://github.com/apache/iceberg/issues/1894#issuecomment-2053813618 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark writes the Iceberg dual partition table to report an error [iceberg]

2024-04-13 Thread via GitHub
github-actions[bot] closed issue #1894: Spark writes the Iceberg dual partition table to report an error URL: https://github.com/apache/iceberg/issues/1894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [0.6.x] Backport #585 [iceberg-python]

2024-04-13 Thread via GitHub
HonahX commented on code in PR #586: URL: https://github.com/apache/iceberg-python/pull/586#discussion_r1564329894 ## pyiceberg/catalog/__init__.py: ## @@ -257,6 +257,12 @@ def delete_data_files(io: FileIO, manifests_to_delete: List[ManifestFile]) -> No deleted

Re: [PR] Spark: Reconcile derived partitioning from source table with target table specs in AddFilesProcedure [iceberg]

2024-04-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #10133: URL: https://github.com/apache/iceberg/pull/10133#discussion_r1564304976 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java: ## @@ -59,9 +59,7 @@ private SparkSchemaUtil() {} * @return a Schema for

[PR] Spark: Simplify SparkSchemaUtil#schemaForTable [iceberg]

2024-04-13 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10137: URL: https://github.com/apache/iceberg/pull/10137 Small refactoring of SparkSchemaUtil#schemaForTable, I noticed when doing #10133 we can just use the existing convert method -- This is an automated message from the Apache Git Service.

Re: [PR] Spark: Reconcile derived partitioning from source table with target table specs in AddFilesProcedure [iceberg]

2024-04-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #10133: URL: https://github.com/apache/iceberg/pull/10133#discussion_r1564299266 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java: ## @@ -59,9 +59,7 @@ private SparkSchemaUtil() {} * @return a Schema for

Re: [PR] Spark: Reconcile derived partitioning from source table with target table specs in AddFilesProcedure [iceberg]

2024-04-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #10133: URL: https://github.com/apache/iceberg/pull/10133#discussion_r1564298758 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -948,6 +948,28 @@ public void testAddFiles

Re: [PR] Add Refs metadata table [iceberg-python]

2024-04-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #602: URL: https://github.com/apache/iceberg-python/pull/602#discussion_r1564265956 ## tests/integration/test_inspect_table.py: ## @@ -266,3 +266,56 @@ def test_inspect_entries_partitioned(spark: SparkSession, session_catalog: Catal

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564246026 ## pyiceberg/io/pyarrow.py: ## @@ -1780,16 +1781,17 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT ) def write_parq

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564241452 ## pyiceberg/io/pyarrow.py: ## @@ -1780,16 +1781,17 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT ) def write_parq

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-04-13 Thread via GitHub
syun64 commented on code in PR #603: URL: https://github.com/apache/iceberg-python/pull/603#discussion_r1564235822 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3411,94 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sch

Re: [PR] Add Refs metadata table [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #602: URL: https://github.com/apache/iceberg-python/pull/602#discussion_r1564223181 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3410,32 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sche

Re: [I] Integration tests performance degradation [iceberg-python]

2024-04-13 Thread via GitHub
kevinjqliu commented on issue #604: URL: https://github.com/apache/iceberg-python/issues/604#issuecomment-2053743307 Some potential optimizations: * parallelize `pytest` execution, this requires that each test can be independently run * un-paratermized some tests, I noticed that tests

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #603: URL: https://github.com/apache/iceberg-python/pull/603#discussion_r1564221753 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3411,94 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sche

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-13 Thread via GitHub
kevinjqliu commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564221724 ## pyiceberg/io/pyarrow.py: ## @@ -1122,12 +1121,12 @@ def project_table( return result -def to_requested_schema(requested_schema: Schema, file_schema:

Re: [I] [feature request] Allow engines to time travel [iceberg-python]

2024-04-13 Thread via GitHub
syun64 commented on issue #600: URL: https://github.com/apache/iceberg-python/issues/600#issuecomment-2053740059 I think this is a great discussion item @kevinjqliu - thank you for raising this. I'm a bit torn between whether we (PyIceberg) should be responsible for creating separat

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-04-13 Thread via GitHub
syun64 commented on code in PR #603: URL: https://github.com/apache/iceberg-python/pull/603#discussion_r1564204302 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3411,94 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sch

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-04-13 Thread via GitHub
syun64 commented on code in PR #603: URL: https://github.com/apache/iceberg-python/pull/603#discussion_r1564204302 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3411,94 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sch

Re: [PR] Support Time Travel in InspectTable.entries [iceberg-python]

2024-04-13 Thread via GitHub
Fokko merged PR #599: URL: https://github.com/apache/iceberg-python/pull/599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564194393 ## tests/integration/test_inspect_table.py: ## @@ -186,8 +185,6 @@ def test_inspect_entries( assert df_lhs == df_rhs, f"Difference in data_fil

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-13 Thread via GitHub
Fokko commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564193928 ## pyiceberg/io/pyarrow.py: ## @@ -1122,12 +1121,12 @@ def project_table( return result -def to_requested_schema(requested_schema: Schema, file_schema: Sche

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on PR #10134: URL: https://github.com/apache/iceberg/pull/10134#issuecomment-2053708559 @manuzhang sure ! Let me update accordingly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
manuzhang commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1564079903 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received an

Re: [PR] [WIP] Integration with Datafusion [iceberg-rust]

2024-04-13 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1563872514 ## Cargo.toml: ## @@ -21,6 +21,7 @@ members = [ "crates/catalog/*", "crates/examples", "crates/iceberg", +"crates/integrations", Review Co

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on PR #10124: URL: https://github.com/apache/iceberg/pull/10124#issuecomment-2053563062 We already have a property to determine if we create tables or not. But it's not directly exposed easily to the users. Let me prepare a PR to improve this. -- This is an automated m

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1563845762 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received any

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

2024-04-13 Thread via GitHub
jbonofre commented on PR #10124: URL: https://github.com/apache/iceberg/pull/10124#issuecomment-2053562342 Sure thing. I can propose a PR during the weekend. Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

2024-04-13 Thread via GitHub
nastra commented on PR #10124: URL: https://github.com/apache/iceberg/pull/10124#issuecomment-2053550821 thanks @mrcnc, I'll take another look on monday. @jbonofre could you also take a look please? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Add stale PRs management [iceberg]

2024-04-13 Thread via GitHub
nastra commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1563814052 ## .github/workflows/stale.yml: ## @@ -47,5 +46,13 @@ jobs: close-issue-message: > This issue has been closed because it has not received any