[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083420820 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/SnapshotTableProcedure.java: ## @@ -93,10 +94,20 @@ public InternalRow[] call(InternalRow a

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083420860 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/MigrateTableProcedure.java: ## @@ -99,7 +100,13 @@ public InternalRow[] call(InternalRow ar

[GitHub] [iceberg] Fokko merged pull request #6641: Build: Bump rich from 13.1.0 to 13.2.0 in /python

Fokko merged PR #6641: URL: https://github.com/apache/iceberg/pull/6641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python

Fokko commented on PR #6640: URL: https://github.com/apache/iceberg/pull/6640#issuecomment-1399445639 Superseded by https://github.com/apache/iceberg/pull/6643 which bumps all the fsspec packages in sync -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [iceberg] Fokko closed pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python

Fokko closed pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python URL: https://github.com/apache/iceberg/pull/6640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [iceberg] dependabot[bot] commented on pull request #6640: Build: Bump adlfs from 2022.11.2 to 2023.1.0 in /python

dependabot[bot] commented on PR #6640: URL: https://github.com/apache/iceberg/pull/6640#issuecomment-1399445664 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

[GitHub] [iceberg] Fokko merged pull request #6639: Build: Bump pandas from 1.5.2 to 1.5.3 in /python

Fokko merged PR #6639: URL: https://github.com/apache/iceberg/pull/6639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083428466 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationE

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083440870 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/MigrateTableProcedure.java: ## @@ -39,7 +39,8 @@ class MigrateTableProcedure extends BasePr

[GitHub] [iceberg] kingeasternsun commented on a diff in pull request #6624: 🎨 Add "parallelism" parameter to "add_files" syscall and MigrateTable, SnapshotTable.

kingeasternsun commented on code in PR #6624: URL: https://github.com/apache/iceberg/pull/6624#discussion_r1083441291 ## api/src/main/java/org/apache/iceberg/actions/MigrateTable.java: ## @@ -50,6 +50,15 @@ default MigrateTable dropBackup() { throw new UnsupportedOperationE

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463683 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463816 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463759 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Docs: java api doc add write data example

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083463953 ## docs/java-api.md: ## @@ -147,6 +147,53 @@ t.newAppend().appendFile(data).commit(); t.commitTransaction(); ``` +### WriteData + +The java api can write data i

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Data: java api add GenericTaskWriter and add write demo to Doc.

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083465798 ## data/src/main/java/org/apache/iceberg/data/GenericTaskWriter.java: ## @@ -0,0 +1,79 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] youngxinler commented on a diff in pull request #6571: Data: java api add GenericTaskWriter and add write demo to Doc.

youngxinler commented on code in PR #6571: URL: https://github.com/apache/iceberg/pull/6571#discussion_r1083466424 ## data/src/main/java/org/apache/iceberg/data/GenericTaskWriter.java: ## @@ -0,0 +1,79 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] youngxinler commented on pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

youngxinler commented on PR #6554: URL: https://github.com/apache/iceberg/pull/6554#issuecomment-1399507127 @RussellSpitzer Can I trouble you if you have time to do a review? About improve Test Coverage of RowGroupFilter Code with Nans. I have made changes. -- This is an automated messag

[GitHub] [iceberg] Fokko commented on a diff in pull request #6590: Python: Add sql command to the CLI

Fokko commented on code in PR #6590: URL: https://github.com/apache/iceberg/pull/6590#discussion_r1083538914 ## python/pyiceberg/io/pyarrow.py: ## @@ -470,6 +472,59 @@ def expression_to_pyarrow(expr: BooleanExpression) -> pc.Expression: return boolean_expression_visit(expr

[GitHub] [iceberg] rubenvdg opened a new pull request, #6644: Python: Add support for static table

rubenvdg opened a new pull request, #6644: URL: https://github.com/apache/iceberg/pull/6644 This PR proposes adding support for static tables (i.e., reading a table directly from a metadata file without using a catalog, see also https://github.com/apache/iceberg/issues/6430). Happy to hear

[GitHub] [iceberg] Fokko opened a new pull request, #6645: Python: Parallelize IO

Fokko opened a new pull request, #6645: URL: https://github.com/apache/iceberg/pull/6645 Alternative for the multithreading part of: https://github.com/apache/iceberg/pull/6590 This uses the ThreadPool approach instead of ThreadPoolExecutor. The ThreadPoolExecutor is more flexi

[GitHub] [iceberg] Fokko commented on a diff in pull request #6644: Python: Add support for static table

Fokko commented on code in PR #6644: URL: https://github.com/apache/iceberg/pull/6644#discussion_r1083543881 ## python/pyiceberg/table/__init__.py: ## @@ -167,6 +169,32 @@ def __eq__(self, other: Any) -> bool: ) +class StaticTable(Table): +"""Load a table direct

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

amogh-jahagirdar commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1083561493 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/RemoveDanglingDeletesSparkAction.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache S

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

amogh-jahagirdar commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1083566960 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/RemoveDanglingDeletesSparkAction.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache S

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #6581: Spark 3.3: Add RemoveDanglingDeletes action

amogh-jahagirdar commented on PR #6581: URL: https://github.com/apache/iceberg/pull/6581#issuecomment-1399644783 @szehon-ho Thanks for the detailed analysis. On the surface it does make sense to combine with existing compaction mechanisms like RewriteDataFiles if the metadata only RemoveDan

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083570497 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -168,34 +175,76 @@ fieldList ; nonRe

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083571839 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572053 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala: ## @@ -61,6 +62,9 @@ case class ExtendedD

[GitHub] [iceberg] github-actions[bot] commented on issue #4581: Spark: Cannot read or write UUID columns

github-actions[bot] commented on issue #4581: URL: https://github.com/apache/iceberg/issues/4581#issuecomment-1399650804 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572437 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572534 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572578 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572778 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083572778 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083573324 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

rdblue commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083573422 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [iceberg] rdblue commented on a diff in pull request #6474: Make it explicit that metrics reporter is required

rdblue commented on code in PR #6474: URL: https://github.com/apache/iceberg/pull/6474#discussion_r1083574777 ## core/src/main/java/org/apache/iceberg/BaseTable.java: ## @@ -48,6 +49,7 @@ public BaseTable(TableOperations ops, String name) { } public BaseTable(TableOperat

[GitHub] [iceberg] rdblue commented on issue #6625: Improve nullability check in Iceberg codebase

rdblue commented on issue #6625: URL: https://github.com/apache/iceberg/issues/6625#issuecomment-1399656418 The reason why we don't use `checkNonNull` is that it throws `NullPointerException`, which mostly makes people think that something tried to dereference it. We use `checkArgument` so

[GitHub] [iceberg] rdblue commented on a diff in pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

rdblue commented on code in PR #6614: URL: https://github.com/apache/iceberg/pull/6614#discussion_r1083575712 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -56,7 +56,6 @@ public R run(Action action, boolean retry) throws E, InterruptedExc C client =

[GitHub] [iceberg] rdblue commented on a diff in pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

rdblue commented on code in PR #6614: URL: https://github.com/apache/iceberg/pull/6614#discussion_r1083575818 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -147,4 +156,10 @@ private void release(C client) { public int poolSize() { return poolSize;

[GitHub] [iceberg] rdblue commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

rdblue commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1399658376 I think that if a catalog is closed, it's reasonable for tables to stop operating as well. The catalog manages its shared resources and if it chooses to share a connection pool with tables

[GitHub] [iceberg] rdblue commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

rdblue commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083582534 ## core/src/main/java/org/apache/iceberg/BaseTransaction.java: ## @@ -551,10 +555,19 @@ public void commit(TableMetadata underlyingBase, TableMetadata metadata) {

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6637: Spark: Spark SQL Extensions for create tag

hililiwei commented on code in PR #6637: URL: https://github.com/apache/iceberg/pull/6637#discussion_r1083582974 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -168,34 +169,61 @@ fieldList ; no

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

ajantha-bhat commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1083604944 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/DistinctCountProcedure.java: ## @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] [iceberg] hililiwei commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

hililiwei commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1399718452 > I think the problem is that the `TableLoader` is taking ownership of the catalog and closing it. That seems incorrect to me. So from that point of view, cloning tables doesn't s

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

hililiwei commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083616314 ## spark/v3.3/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4: ## @@ -168,34 +175,76 @@ fieldList ; no

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6617: Spark: Spark SQL Extensions for create branch

hililiwei commented on code in PR #6617: URL: https://github.com/apache/iceberg/pull/6617#discussion_r1083616874 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateBranch.java: ## @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software F

[GitHub] [iceberg] arminnajafi opened a new pull request, #6646: Implement Support for DynamoDB Catalog

arminnajafi opened a new pull request, #6646: URL: https://github.com/apache/iceberg/pull/6646 Fixes #6541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

[GitHub] [iceberg] stevenzwu commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

stevenzwu commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1399807014 > I think the problem is that the TableLoader is taking ownership of the catalog and closing it. That seems incorrect to me. Yes, that will be bigger discussion. > Alterna

[GitHub] [iceberg] stevenzwu merged pull request #6635: Flink: add table setter to FLIP-27 IcebergSource#Builder.

stevenzwu merged PR #6635: URL: https://github.com/apache/iceberg/pull/6635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6634: Core, API: Fix for tracking intermediate snapshots when a transaction spans multiple branches

amogh-jahagirdar commented on code in PR #6634: URL: https://github.com/apache/iceberg/pull/6634#discussion_r1083658719 ## core/src/main/java/org/apache/iceberg/BaseTransaction.java: ## @@ -551,10 +555,19 @@ public void commit(TableMetadata underlyingBase, TableMetadata metadat

[GitHub] [iceberg] XN137 commented on pull request #6629: Build: Fix minor error-prone warnings

XN137 commented on PR #6629: URL: https://github.com/apache/iceberg/pull/6629#issuecomment-1399881912 should we raise the severity of the `StringSplitter` check to `ERROR` like [here](https://github.com/apache/iceberg/blob/4491e7d6d6911ca0ecf129d7528dda2717f8c938/baseline.gradle#L70) ? t

[GitHub] [iceberg] pvary commented on pull request #6614: Flink:fix flink streaming query problem [ Cannot get a client from a closed pool]

pvary commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1399896611 > The problem for ContinuousSplitPlannerImpl of FLIP-27 IcebergSource is that it uses the Table as non read-only, as it needs to refresh the table. `Table` API assumes that you can ha

[GitHub] [iceberg] ajantha-bhat commented on pull request #6629: Build: Fix minor error-prone warnings

ajantha-bhat commented on PR #6629: URL: https://github.com/apache/iceberg/pull/6629#issuecomment-1399900490 > should we raise the severity of the StringSplitter check to ERROR like [here](https://github.com/apache/iceberg/blob/4491e7d6d6911ca0ecf129d7528dda2717f8c938/baseline.gradle#L70) ?

[GitHub] [iceberg] XN137 commented on pull request #6629: Build: Fix minor error-prone warnings

XN137 commented on PR #6629: URL: https://github.com/apache/iceberg/pull/6629#issuecomment-1399910688 if these warnings are worth fixing, shouldnt they also be worth preventing? automatic prevention is usually cheaper than manually observing and fixing future warnings. this also ha