Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

nastra commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1565260333 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkSource.java: ## @@ -60,7 +60,7 @@ protected List runWithFilter(Expression filter, String s

Re: [PR] Migrate non TestBase related and Data classes in Flink [iceberg]

tomtongue commented on PR #10130: URL: https://github.com/apache/iceberg/pull/10130#issuecomment-2055673777 Sure. Will submit a PR for backporting. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Add Refs metadata table [iceberg-python]

geruh commented on code in PR #602: URL: https://github.com/apache/iceberg-python/pull/602#discussion_r1565252853 ## tests/integration/test_inspect_table.py: ## @@ -266,3 +266,56 @@ def test_inspect_entries_partitioned(spark: SparkSession, session_catalog: Catal assert d

Re: [PR] Migrate non TestBase related and Data classes in Flink [iceberg]

nastra merged PR #10130: URL: https://github.com/apache/iceberg/pull/10130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Migrate non TestBase related and Data classes in Flink [iceberg]

nastra commented on PR #10130: URL: https://github.com/apache/iceberg/pull/10130#issuecomment-2055647950 changes LGTM, can you please backport these changes to earlier Flink versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Migrate non TestBase related and Data classes in Flink [iceberg]

nastra commented on code in PR #10130: URL: https://github.com/apache/iceberg/pull/10130#discussion_r1565251166 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/data/TestRowProjection.java: ## @@ -37,19 +41,17 @@ import org.apache.iceberg.relocated.com.google.common.

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

nastra commented on code in PR #10124: URL: https://github.com/apache/iceberg/pull/10124#discussion_r1565243349 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -137,6 +137,10 @@ public void initialize(String name, Map properties) { this.connections

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

nastra commented on code in PR #10124: URL: https://github.com/apache/iceberg/pull/10124#discussion_r1565243207 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -1020,4 +1062,36 @@ private void initLegacySchema(String jdbcUrl) throws SQLException {

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

nastra commented on code in PR #10124: URL: https://github.com/apache/iceberg/pull/10124#discussion_r1565239656 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -1020,4 +1062,36 @@ private void initLegacySchema(String jdbcUrl) throws SQLException {

Re: [PR] Add Refs metadata table [iceberg-python]

geruh commented on code in PR #602: URL: https://github.com/apache/iceberg-python/pull/602#discussion_r1565238973 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3410,32 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sche

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

ajantha-bhat commented on code in PR #10124: URL: https://github.com/apache/iceberg/pull/10124#discussion_r1565210846 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -137,6 +137,10 @@ public void initialize(String name, Map properties) { this.connec

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

jbonofre commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565208210 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -56,26 +67,36 @@ public R run(Action action, boolean retry) throws E, InterruptedExc C cli

Re: [I] In case of Positional Deletes "file_path" in lowerbound & upperbound do not have full file_path Buffer [iceberg]

agrawalreetika closed issue #10064: In case of Positional Deletes "file_path" in lowerbound & upperbound do not have full file_path Buffer URL: https://github.com/apache/iceberg/issues/10064 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

ajantha-bhat commented on PR #10140: URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2055350705 cc: @jbonofre -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Core: Fix logging table name in scanning metadata table [iceberg]

manuzhang opened a new pull request, #10141: URL: https://github.com/apache/iceberg/pull/10141 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

lintingbin commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2054624104 @szehon-ho @stevenzwu Can you help me take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] DeleteOrphanFiles or ExpireSnapshots outofmemory [iceberg]

manuzhang commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-2054454593 @RussellSpitzer I wonder where we are now for this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Update site docs to have a CustomCatalog page for python [iceberg-python]

github-actions[bot] closed issue #15: Update site docs to have a CustomCatalog page for python URL: https://github.com/apache/iceberg-python/issues/15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Support fan-out reads in PyIceberg [iceberg-python]

github-actions[bot] commented on issue #16: URL: https://github.com/apache/iceberg-python/issues/16#issuecomment-2054440487 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Update site docs to have a CustomCatalog page for python [iceberg-python]

github-actions[bot] commented on issue #15: URL: https://github.com/apache/iceberg-python/issues/15#issuecomment-2054440623 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] The spark's remove_orphan_files procedure cannot expire the orphan files that located in remote object storage services [iceberg]

github-actions[bot] commented on issue #2525: URL: https://github.com/apache/iceberg/issues/2525#issuecomment-2054433410 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Partitioning on sensitive (encrypted) columns [iceberg]

github-actions[bot] commented on issue #2513: URL: https://github.com/apache/iceberg/issues/2513#issuecomment-2054433255 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink CDC | OOM during initial snapshot [iceberg]

github-actions[bot] commented on issue #2504: URL: https://github.com/apache/iceberg/issues/2504#issuecomment-2054433062 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Internal Tests Cannot us ORC or Avro Datasource Writers [iceberg]

github-actions[bot] commented on issue #2360: URL: https://github.com/apache/iceberg/issues/2360#issuecomment-2054432411 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Internal Tests Cannot us ORC or Avro Datasource Writers [iceberg]

github-actions[bot] closed issue #2360: Internal Tests Cannot us ORC or Avro Datasource Writers URL: https://github.com/apache/iceberg/issues/2360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Iceberg Extensions Parser Should Allow Generic Expressions as Call Args [iceberg]

github-actions[bot] closed issue #2348: Iceberg Extensions Parser Should Allow Generic Expressions as Call Args URL: https://github.com/apache/iceberg/issues/2348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] TimeZone‘s ERROR on testInsertSupportedTypes [iceberg]

github-actions[bot] commented on issue #2356: URL: https://github.com/apache/iceberg/issues/2356#issuecomment-2054432249 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Iceberg Extensions Parser Should Allow Generic Expressions as Call Args [iceberg]

github-actions[bot] commented on issue #2348: URL: https://github.com/apache/iceberg/issues/2348#issuecomment-2054432064 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] trion very slow [iceberg]

github-actions[bot] closed issue #2351: trion very slow URL: https://github.com/apache/iceberg/issues/2351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Support fan-out reads in PyIceberg [iceberg-python]

github-actions[bot] closed issue #16: Support fan-out reads in PyIceberg URL: https://github.com/apache/iceberg-python/issues/16 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Predicate pushdown not visible in Spark plan [iceberg]

github-actions[bot] commented on issue #2517: URL: https://github.com/apache/iceberg/issues/2517#issuecomment-205445 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Stack map does not match the one at exception handler [iceberg]

github-actions[bot] commented on issue #2507: URL: https://github.com/apache/iceberg/issues/2507#issuecomment-2054433167 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg supports Tencent COS [iceberg]

github-actions[bot] commented on issue #2498: URL: https://github.com/apache/iceberg/issues/2498#issuecomment-2054432969 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] hive cannot get data from iceberg table [iceberg]

github-actions[bot] closed issue #2357: hive cannot get data from iceberg table URL: https://github.com/apache/iceberg/issues/2357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] TimeZone‘s ERROR on testInsertSupportedTypes [iceberg]

github-actions[bot] closed issue #2356: TimeZone‘s ERROR on testInsertSupportedTypes URL: https://github.com/apache/iceberg/issues/2356 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] hive cannot get data from iceberg table [iceberg]

github-actions[bot] commented on issue #2357: URL: https://github.com/apache/iceberg/issues/2357#issuecomment-2054432332 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] trion very slow [iceberg]

github-actions[bot] commented on issue #2351: URL: https://github.com/apache/iceberg/issues/2351#issuecomment-2054432150 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Migrate non TestBase related and Data classes in Flink [iceberg]

tomtongue commented on PR #10130: URL: https://github.com/apache/iceberg/pull/10130#issuecomment-2054279066 @nastra thanks for the review. Reflected your feedback and migrate other classes. Could you review? -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Add stale PRs management [iceberg]

ajantha-bhat commented on code in PR #10134: URL: https://github.com/apache/iceberg/pull/10134#discussion_r1565022616 ## .github/workflows/stale.yml: ## @@ -33,12 +33,11 @@ jobs: steps: - uses: actions/stale@v9.0.0 with: + # stale issues Review Com

Re: [PR] Core: Fix retry behavior for Jdbc Client [iceberg]

amogh-jahagirdar commented on PR #7561: URL: https://github.com/apache/iceberg/pull/7561#issuecomment-2054257138 @jean-humann @cccs-br @cccs-eric @cccs-jc https://github.com/apache/iceberg/pull/10140 this is a draft PR for having a configurable set of retry codes. I'll need to write some t

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565013391 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public J

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565013391 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public J

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

amogh-jahagirdar commented on PR #10140: URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2054248088 Need to add tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]

amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565006119 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -56,26 +67,36 @@ public R run(Action action, boolean retry) throws E, InterruptedExc

[PR] Retry connections in JDBC catalog [iceberg]

amogh-jahagirdar opened a new pull request, #10140: URL: https://github.com/apache/iceberg/pull/10140 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564892433 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestChangeLogTable.java: ## @@ -98,7 +98,7 @@ public void before() { @Override public void clean(

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564937403 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestIcebergConnector.java: ## @@ -257,20 +257,12 @@ public void testCreateTableUnderDefaultDatabase() {

Re: [I] Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. [iceberg]

flisboac commented on issue #9650: URL: https://github.com/apache/iceberg/issues/9650#issuecomment-2054171388 Well, now errors are happening on a `MERGE INTO` as well. Usage is the same as what the OP reported. Also, because I'm using PySpark, a more detailed error report is somewhat hidden

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564895113 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -69,7 +69,9 @@ public class FlinkCatalogFactory implements CatalogFacto

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564892433 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestChangeLogTable.java: ## @@ -98,7 +98,7 @@ public void before() { @Override public void clean(

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564895113 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -69,7 +69,9 @@ public class FlinkCatalogFactory implements CatalogFacto

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564895113 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java: ## @@ -69,7 +69,9 @@ public class FlinkCatalogFactory implements CatalogFacto

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

rodmeneses commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1564892433 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestChangeLogTable.java: ## @@ -98,7 +98,7 @@ public void before() { @Override public void clean(

Re: [I] Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. [iceberg]

flisboac commented on issue #9650: URL: https://github.com/apache/iceberg/issues/9650#issuecomment-2054163238 Just to contribute to this report, I can replicate this error with a table that's configured as Merge-On-Read for inserts and updates, and as Copy-on-Write for Deletes. It's a CDC t

Re: [I] Implement rolling manifest-writers [iceberg-python]

felixscherz commented on issue #596: URL: https://github.com/apache/iceberg-python/issues/596#issuecomment-2054140100 Hi, I would like to give this a shot if no one else is currently working on this:) I see the java implementation has a `RollingManifestWriter` so I'd start there. -- T

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

kevinjqliu commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564830649 ## pyiceberg/io/pyarrow.py: ## @@ -1769,27 +1769,33 @@ def data_file_statistics_from_parquet_metadata( def write_file(io: FileIO, table_metadata: TableMeta

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

kevinjqliu commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1564828814 ## pyiceberg/io/pyarrow.py: ## @@ -1769,27 +1769,33 @@ def data_file_statistics_from_parquet_metadata( def write_file(io: FileIO, table_metadata: TableMeta

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2054129400 Thank you @kevinjqliu ! Do you know how to read parquet file with unified schema in pyarrow? I successfully merged schemas: ``` t1 = pq.read_table("data/1.parq

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

ZENOTME commented on PR #277: URL: https://github.com/apache/iceberg-rust/pull/277#issuecomment-2054118443 I think this PR is ready to go now. cc @liurenjie1024 @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [WIP] Integration with Datafusion [iceberg-rust]

tshauck commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1564807159 ## crates/integrations/src/datafusion/schema.rs: ## @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] Spark: Simplify SparkSchemaUtil#schemaForTable [iceberg]

amogh-jahagirdar merged PR #10137: URL: https://github.com/apache/iceberg/pull/10137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Simplify SparkSchemaUtil#schemaForTable [iceberg]

amogh-jahagirdar commented on PR #10137: URL: https://github.com/apache/iceberg/pull/10137#issuecomment-2054066583 Thanks for reviewing @Fokko ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] chore(deps): Bump peaceiris/actions-gh-pages from 3.9.3 to 4.0.0 [iceberg-rust]

dependabot[bot] opened a new pull request, #333: URL: https://github.com/apache/iceberg-rust/pull/333 Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3.9.3 to 4.0.0. Release notes Sourced from https://github.com/peaceiris/actions-gh-pages/rel

[PR] chore(deps): Bump peaceiris/actions-mdbook from 1 to 2 [iceberg-rust]

dependabot[bot] opened a new pull request, #332: URL: https://github.com/apache/iceberg-rust/pull/332 Bumps [peaceiris/actions-mdbook](https://github.com/peaceiris/actions-mdbook) from 1 to 2. Release notes Sourced from https://github.com/peaceiris/actions-mdbook/releases";>peacei

Re: [PR] Core: Add property to disable table initialization for JdbcCatalog [iceberg]

jbonofre commented on PR #10124: URL: https://github.com/apache/iceberg/pull/10124#issuecomment-2053962700 This PR is ok for me. I will prepare another pr to provide create statement as sql file and document it. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Build: Bump org.springframework:spring-web from 5.3.33 to 5.3.34 [iceberg]

Fokko merged PR #10139: URL: https://github.com/apache/iceberg/pull/10139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.31 [iceberg]

Fokko merged PR #10138: URL: https://github.com/apache/iceberg/pull/10138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa