[I] iceberg docker-compose quickstart fail [iceberg]

2024-11-13 Thread via GitHub
HelloWorldwangjiacheng opened a new issue, #11544: URL: https://github.com/apache/iceberg/issues/11544 ### Apache Iceberg version 1.5.0 ### Query engine Spark ### Please describe the bug šŸž Background: I just want to make a quick-start spark-iceberg under do

[I] Flink `write-parallelism` causes distribution-mode: RANGE to not work properly [iceberg]

2024-11-13 Thread via GitHub
huyuanfeng2018 opened a new issue, #11543: URL: https://github.com/apache/iceberg/issues/11543 ### Apache Iceberg version 1.7.0 (latest release) ### Query engine Flink ### Please describe the bug šŸž When we set write-parallelism, if the parallelism of writing

Re: [I] Flink: Not Writing [iceberg]

2024-11-13 Thread via GitHub
belldoor commented on issue #8916: URL: https://github.com/apache/iceberg/issues/8916#issuecomment-2475626871 Same here, No errors are displayed at all, only the table is created and data is not written. You can easily reproduce from this code. ```py from pyflink.datastr

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1841704909 ## open-api/rest-catalog-open-api.yaml: ## @@ -3142,6 +3162,10 @@ components: type: object additionalProperties: type: string +

Re: [PR] Data, Flink, Spark: Test deletes with format-version=3 [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11538: URL: https://github.com/apache/iceberg/pull/11538#discussion_r1841692014 ## data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java: ## @@ -82,12 +83,15 @@ public abstract class DeleteReadTests { @Parameter protected FileFormat

Re: [PR] Spark: Test deletes with format-version=3 [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11538: URL: https://github.com/apache/iceberg/pull/11538#discussion_r1841683046 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java: ## @@ -99,16 +100,21 @@ public class TestSparkReaderDeletes extends Del

Re: [PR] Spark: Test deletes with format-version=3 [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11538: URL: https://github.com/apache/iceberg/pull/11538#discussion_r1841682548 ## mr/src/test/java/org/apache/iceberg/mr/TestInputFormatReaderDeletes.java: ## @@ -49,18 +49,18 @@ public class TestInputFormatReaderDeletes extends DeleteReadTests

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
nastra commented on PR #11419: URL: https://github.com/apache/iceberg/pull/11419#issuecomment-2475577208 thanks @rocco408, this is almost ready to go, just had a few more minor comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1841671746 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2.java: ## @@ -244,4 +247,132 @@ public void testMergeSchemaSparkConfiguration()

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1841670685 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2.java: ## @@ -209,4 +213,132 @@ public void testWriteWithCaseSensitiveOption() t

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1841669608 ## core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java: ## @@ -180,6 +180,23 @@ private void updateColumn(Types.NestedField field, Types.NestedField

[PR] [DRAFT]: Adjust site links to absolute from site_url [iceberg]

2024-11-13 Thread via GitHub
bitsondatadev opened a new pull request, #9887: URL: https://github.com/apache/iceberg/pull/9887 Avoid internal [MkDocs relative linking](https://www.mkdocs.org/user-guide/writing-your-docs/#linking-to-pages) using the [mkdocs-site-urls](https://github.com/OctoPrint/mkdocs-site-urls) plugi

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
pvary commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841614061 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
pvary commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841605909 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
pvary commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841611540 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -276,29 +215,25 @@ RewriteDataFilesCommitManager commitManage

Re: [PR] Spark partial limit push down [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1841589907 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java: ## @@ -48,6 +50,17 @@ abstract class BaseRowReader extends BaseReader taskGro

Re: [PR] Spark partial limit push down [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1841589128 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBatch

Re: [I] Is `pyiceberg.Table` thread-safe? [iceberg-python]

2024-11-13 Thread via GitHub
chengchengpei closed issue #1305: Is `pyiceberg.Table` thread-safe? URL: https://github.com/apache/iceberg-python/issues/1305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] TestDataFrameWrites#testFaultToleranceOnWrite failed due to exception in cleaning up temporary directory [iceberg]

2024-11-13 Thread via GitHub
manuzhang commented on issue #10172: URL: https://github.com/apache/iceberg/issues/10172#issuecomment-2475390047 This looks to be fixed by #11470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] TestDataFrameWrites#testFaultToleranceOnWrite failed due to exception in cleaning up temporary directory [iceberg]

2024-11-13 Thread via GitHub
manuzhang closed issue #10172: TestDataFrameWrites#testFaultToleranceOnWrite failed due to exception in cleaning up temporary directory URL: https://github.com/apache/iceberg/issues/10172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-11-13 Thread via GitHub
FANNG1 commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1841531249 ## open-api/rest-catalog-open-api.yaml: ## @@ -3142,6 +3162,10 @@ components: type: object additionalProperties: type: string +

Re: [I] Direct memory leaks when reading parquet files containing interleaving plain/dictionary pages [iceberg]

2024-11-13 Thread via GitHub
CodingJun commented on issue #11533: URL: https://github.com/apache/iceberg/issues/11533#issuecomment-2475362579 > @CodingJun Are you able to check if the problem persists in 1.7.0? Yes, I have checked, and there is still a problem in 1.7.0. -- This is an automated message from the

[PR] Bug Fix: `metadata_location` optional [iceberg-python]

2024-11-13 Thread via GitHub
sungwy opened a new pull request, #1321: URL: https://github.com/apache/iceberg-python/pull/1321 While testing out the iceberg-rest-catalog docker image, I ran into the following error when parsing the following example TableResponse that is created in `tests/integration/test_writes/test_w

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
rocco408 commented on PR #11419: URL: https://github.com/apache/iceberg/pull/11419#issuecomment-2475221160 Thanks @nastra, I think I addressed all your suggestions, cheers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-13 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1841381304 ## tests/integration/test_writes/test_writes.py: ## @@ -1448,3 +1450,100 @@ def test_rewrite_manifest_after_partition_evolution(session_catalog: Catalog) ->

Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]

2024-11-13 Thread via GitHub
sungwy commented on issue #1284: URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2474284318 > If Iceberg Schema is given, it will use its original assigned ID. If PyArrow Schemas is given, it will be converted to Iceberg Schemas and have IDs assigned. @kevinjq

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-11-13 Thread via GitHub
danielcweeks commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1841393266 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java: ## @@ -59,20 +87,29 @@ protected static Object[][] parameters() { }

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-11-13 Thread via GitHub
danielcweeks commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1841394148 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java: ## @@ -59,20 +87,29 @@ protected static Object[][] parameters() { }

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-11-13 Thread via GitHub
danielcweeks commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1841393266 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java: ## @@ -59,20 +87,29 @@ protected static Object[][] parameters() { }

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
rocco408 commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1841387395 ## core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java: ## @@ -163,8 +164,7 @@ private void updateColumn(Types.NestedField field, Types.NestedField

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-13 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2475142299 Thank you for the review @kevinjqliu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Remove Dependency on Hadoop's Filesystem Class from Remove Orphan Files [iceberg]

2024-11-13 Thread via GitHub
amogh-jahagirdar commented on issue #11541: URL: https://github.com/apache/iceberg/issues/11541#issuecomment-2475141314 https://github.com/apache/iceberg/pull/7914 should address this, it'd be great if someone could take this forward. -- This is an automated message from the Apache Git Se

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-13 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1841381304 ## tests/integration/test_writes/test_writes.py: ## @@ -1448,3 +1450,100 @@ def test_rewrite_manifest_after_partition_evolution(session_catalog: Catalog) ->

Re: [PR] Spark: revert delete procedure [iceberg]

2024-11-13 Thread via GitHub
bryanck commented on PR #11084: URL: https://github.com/apache/iceberg/pull/11084#issuecomment-2475104270 Replying to keep this aliveā€¦ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Data loss in the Incremental Co-operative Mode of Rebalancing [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11289: URL: https://github.com/apache/iceberg/pull/11289#issuecomment-2475084391 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Nessie: respect the nearest namespace's `location` property when creating a table or view [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11215: URL: https://github.com/apache/iceberg/pull/11215#issuecomment-2475084336 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] PoC: Add Variant type support in Iceberg [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11201: URL: https://github.com/apache/iceberg/pull/11201#issuecomment-2475084283 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Update Examples to Replace Hadoop Catalog with JDBC Catalog [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11285: URL: https://github.com/apache/iceberg/pull/11285#issuecomment-2475084364 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Always update table metadata when `refresh` is called [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11194: URL: https://github.com/apache/iceberg/pull/11194#issuecomment-2475084227 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Compatible with Spark4 ļ¼ˆupgrade antlr4 to version 4.13.1 Compatible with jdk17Ā  ļ¼‰ [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11204: URL: https://github.com/apache/iceberg/pull/11204#issuecomment-2475084310 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] OpenAPI: Add planning-mode to loadTable response [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11156: URL: https://github.com/apache/iceberg/pull/11156#issuecomment-2475084133 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Docs: Add Bigquery Iceberg documentation, Update MRAP endpoint and add more docs [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11159: URL: https://github.com/apache/iceberg/pull/11159#issuecomment-2475084167 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] FIX: Exception Handling in AWS Glue renameTable Method [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11165: URL: https://github.com/apache/iceberg/pull/11165#issuecomment-2475084200 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Spark: revert delete procedure [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11084: URL: https://github.com/apache/iceberg/pull/11084#issuecomment-2475084081 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Core: Add support for view-override property in catalog [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11200: URL: https://github.com/apache/iceberg/pull/11200#issuecomment-2475084262 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Kafka Connect: add option to force columns to lowercase [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #11100: URL: https://github.com/apache/iceberg/pull/11100#issuecomment-2475084112 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-11-13 Thread via GitHub
github-actions[bot] commented on PR #10877: URL: https://github.com/apache/iceberg/pull/10877#issuecomment-2475083883 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatā€™s incorrect or this pul

[PR] Bump getdaft from 0.3.12 to 0.3.13 [iceberg-python]

2024-11-13 Thread via GitHub
dependabot[bot] opened a new pull request, #1320: URL: https://github.com/apache/iceberg-python/pull/1320 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.3.12 to 0.3.13. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

[PR] Bump mkdocs-material from 9.5.43 to 9.5.44 [iceberg-python]

2024-11-13 Thread via GitHub
dependabot[bot] opened a new pull request, #1319: URL: https://github.com/apache/iceberg-python/pull/1319 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.43 to 9.5.44. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-13 Thread via GitHub
wypoon commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1841283880 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -55,8 +55,14 @@ public ColumnarBatchReader(List> readers) {

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841263893 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
nastra commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1839878542 ## core/src/test/java/org/apache/iceberg/TestSchemaUnionByFieldName.java: ## @@ -303,13 +303,33 @@ public void testTypePromoteFloatToDouble() { } @Test - publ

[PR] use the right file name [iceberg-go]

2024-11-13 Thread via GitHub
kevinjqliu opened a new pull request, #203: URL: https://github.com/apache/iceberg-go/pull/203 While running `release_rc.sh` for RC2, I noticed that the .asc file was not generated. There's a bug in the script similar to #199 and #202 where the filename does not include the RC version.

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-13 Thread via GitHub
wypoon commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1841283880 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -55,8 +55,14 @@ public ColumnarBatchReader(List> readers) {

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2024-11-13 Thread via GitHub
bryanck commented on code in PR #11504: URL: https://github.com/apache/iceberg/pull/11504#discussion_r1841274278 ## azure/src/main/java/org/apache/iceberg/azure/AzureProperties.java: ## @@ -93,7 +93,7 @@ public void applyClientConfiguration(String account, DataLakeFileSystemCli

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2024-11-13 Thread via GitHub
bryanck commented on code in PR #11504: URL: https://github.com/apache/iceberg/pull/11504#discussion_r1841269900 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSLocation.java: ## @@ -30,14 +30,21 @@ * * Locations follow a URI like structure to identify resources

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841263893 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on PR #11513: URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2474919672 For easier review it would be great if you could highlight the changes made. I see the note on RewritePlanResult but i'm not sure where this class came from. It is unclear

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841271760 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -276,29 +215,25 @@ RewriteDataFilesCommitManager com

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841270029 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -168,89 +160,36 @@ public RewriteDataFiles.Result ex

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2024-11-13 Thread via GitHub
bryanck commented on code in PR #11504: URL: https://github.com/apache/iceberg/pull/11504#discussion_r1841268590 ## azure/src/main/java/org/apache/iceberg/azure/AzureProperties.java: ## @@ -93,7 +93,7 @@ public void applyClientConfiguration(String account, DataLakeFileSystemCli

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841265128 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841227042 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841247322 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]

2024-11-13 Thread via GitHub
kevinjqliu opened a new issue, #1317: URL: https://github.com/apache/iceberg-python/issues/1317 ### Apache Iceberg version None ### Please describe the bug šŸž From devlist, https://lists.apache.org/thread/lvtx80c2b47ckpfrylo4ot5o00nfb5r4 ``` When attempting to u

Re: [PR] Fix `version` and `rc` in `rc.yml` [iceberg-go]

2024-11-13 Thread via GitHub
RussellSpitzer merged PR #202: URL: https://github.com/apache/iceberg-go/pull/202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Fix `version` and `rc` in `rc.yml` [iceberg-go]

2024-11-13 Thread via GitHub
RussellSpitzer commented on PR #202: URL: https://github.com/apache/iceberg-go/pull/202#issuecomment-2474842101 Thanks @kevinjqliu for the PR and @zeroshade for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[I] remove warnings about "Catalog identifiers" if one is not provided [iceberg-python]

2024-11-13 Thread via GitHub
kevinjqliu opened a new issue, #1318: URL: https://github.com/apache/iceberg-python/issues/1318 ### Apache Iceberg version None ### Please describe the bug šŸž ### Apache Iceberg version None ### Please describe the bug šŸž From devlist, https://lists.ap

Re: [PR] Bump deptry from 0.20.0 to 0.21.0 [iceberg-python]

2024-11-13 Thread via GitHub
Fokko merged PR #1313: URL: https://github.com/apache/iceberg-python/pull/1313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474802670 Thanks @amogh-jahagirdar @szehon-ho @Fokko @dramaticlly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] assumeThat(format).isEqualTo(parquet) -> assumeThat(format).isEqualTo(FileFormat.PARQUET) [iceberg]

2024-11-13 Thread via GitHub
szehon-ho commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474573217 So strange, do you know why it aborts instead of fails? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
amogh-jahagirdar commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474791454 Thanks @huaxingao , and @szehon-ho @Fokko @dramaticlly for reviewing the changes -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
amogh-jahagirdar merged PR #11540: URL: https://github.com/apache/iceberg/pull/11540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474711816 > actually wdyt to change it to assertThat? In this test suite, the file format could also be [ORC or AVRO](https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/test

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1841100645 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -55,8 +55,14 @@ public ColumnarBatchReader(List> readers

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474655039 > So strange, do you know why it aborts instead of fails? I think the purpose of `assumeThat` is to determine whether the conditions are suitable for a test to continue. If an a

Re: [PR] Spark: Use assumeThat(format).isEqualTo(FileFormat.PARQUET) instead of assumeThat(format).isEqualTo(parquet) [iceberg]

2024-11-13 Thread via GitHub
szehon-ho commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474666252 Ah got it, actually wdyt to change it to assertThat? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Fix `version` and `rc` in `rc.yml` [iceberg-go]

2024-11-13 Thread via GitHub
kevinjqliu commented on code in PR #202: URL: https://github.com/apache/iceberg-go/pull/202#discussion_r1841065941 ## .github/workflows/rc.yml: ## @@ -42,17 +42,25 @@ jobs: - name: Prepare for tag if: github.ref_type == 'tag' run: | - version=${

[PR] Fix `version` and `rc` in `rc.yml` [iceberg-go]

2024-11-13 Thread via GitHub
kevinjqliu opened a new pull request, #202: URL: https://github.com/apache/iceberg-go/pull/202 Follow up to #199. For the RC workflow running on a tag, there was an issue resulting in the wrong filename used for the `tar.gz` file, for example, `apache-iceberg-go-0.1.0-rc1-rc1.tar.gz`. No

Re: [PR] assumeThat(format).isEqualTo(parquet) -> assumeThat(format).isEqualTo(FileFormat.PARQUET) [iceberg]

2024-11-13 Thread via GitHub
szehon-ho commented on PR #11540: URL: https://github.com/apache/iceberg/pull/11540#issuecomment-2474580572 Can you fix the pr description like : ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Bump getdaft from 0.3.11 to 0.3.12 [iceberg-python]

2024-11-13 Thread via GitHub
Fokko merged PR #1316: URL: https://github.com/apache/iceberg-python/pull/1316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[PR] assumeThat(format).isEqualTo(parquet) -> assumeThat(format).isEqualTo(FileFormat.PARQUET) [iceberg]

2024-11-13 Thread via GitHub
huaxingao opened a new pull request, #11540: URL: https://github.com/apache/iceberg/pull/11540 With assertj-core-3.26.3.jar, assumeThat(format).isEqualTo("parquet") is failed and the test is silently aborted. https://github.com/user-attachments/assets/2a1caccd-58ac-4cda-9c86-1434bc07cb77"

Re: [PR] Spark 3.4: Support Spark Column Stats [iceberg]

2024-11-13 Thread via GitHub
huaxingao commented on PR #11532: URL: https://github.com/apache/iceberg/pull/11532#issuecomment-2474477341 @saitharun15 https://github.com/apache/iceberg/pull/11035 fixed SparkScan to only read Apache DataSketches. Could you please back-port these changes too? -- This is an automated mes

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840963359 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -107,6 +107,7 @@ public void basicCreateView() { assertThat(view.currentVersion

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840963359 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -107,6 +107,7 @@ public void basicCreateView() { assertThat(view.currentVersion

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840963359 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -107,6 +107,7 @@ public void basicCreateView() { assertThat(view.currentVersion

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840945935 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -85,6 +86,7 @@ public String name() { @Override public void initialize(Stri

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-13 Thread via GitHub
BlakeOrth commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2474390321 Wow, I did not expect my comment digging up a nearly year old issue to result in this much discussion! I likely have less skin in the game than any of the maintainers here

Re: [PR] Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner [iceberg]

2024-11-13 Thread via GitHub
mayankvadariya commented on PR #10858: URL: https://github.com/apache/iceberg/pull/10858#issuecomment-2474384531 hi @nastra, I've created https://github.com/apache/iceberg/issues/11539 with reproduction steps. -- This is an automated message from the Apache Git Service. To respond to the

[I] Nested namespace support is broken in 1.7.0 [iceberg]

2024-11-13 Thread via GitHub
mayankvadariya opened a new issue, #11539: URL: https://github.com/apache/iceberg/issues/11539 ### Apache Iceberg version 1.7.0 (latest release) ### Query engine Trino ### Please describe the bug šŸž Nested namespace request to Iceberg catalog server is sent i

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840946761 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -94,6 +96,11 @@ public void initialize(String name, Map properties) { close

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1840549260 ## core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java: ## @@ -180,6 +180,17 @@ private void updateColumn(Types.NestedField field, Types.Nest

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1840938983 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -29,6 +29,7 @@ private CatalogProperties() {} public static final String WAREHOUSE_LO

Re: [PR] Spark: Test deletes with format-version=3 [iceberg]

2024-11-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #11538: URL: https://github.com/apache/iceberg/pull/11538#discussion_r1840930383 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java: ## @@ -99,16 +100,21 @@ public class TestSparkReaderDeletes ex

Re: [PR] Core, Flink, Spark: Verify maintenance actions with DVs [iceberg]

2024-11-13 Thread via GitHub
aokolnychyi commented on code in PR #11485: URL: https://github.com/apache/iceberg/pull/11485#discussion_r1840256285 ## core/src/main/java/org/apache/iceberg/BaseContentScanTask.java: ## @@ -82,7 +83,7 @@ public long start() { @Override public long length() { -return

Re: [PR] Fix verification script, `verify_rc.sh` [iceberg-go]

2024-11-13 Thread via GitHub
Fokko merged PR #199: URL: https://github.com/apache/iceberg-go/pull/199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark partial limit push down [iceberg]

2024-11-13 Thread via GitHub
RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840791166 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

2024-11-13 Thread via GitHub
aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840750027 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBat

Re: [I] Why shouldn't we return an `UnboundPartitionSpec` instead? [iceberg-rust]

2024-11-13 Thread via GitHub
c-thiel commented on issue #694: URL: https://github.com/apache/iceberg-rust/issues/694#issuecomment-2474153227 I think I have most of my motivation in https://github.com/apache/iceberg-rust/pull/645#issue-2543573501 and the comments referenced above. I follow the argument from Renjie.

Re: [I] Schema: Allow field name `foo.bar` even if struct foo->bar is present [iceberg-rust]

2024-11-13 Thread via GitHub
c-thiel commented on issue #591: URL: https://github.com/apache/iceberg-rust/issues/591#issuecomment-2474101258 This is not so much about this specific use case - which I also don't care about much either, but about having two different representations for the same entity. Took me a while t

  1   2   >