dependabot[bot] opened a new pull request, #8786: URL: https://github.com/apache/iceberg/pull/8786
Bumps [io.delta:delta-core_2.12](https://github.com/delta-io/delta) from 2.2.0 to 2.4.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-core_2.12's releases</a>.</em></p> <blockquote> <h2>Delta Lake 2.4.0</h2> <p>We are excited to announce the release of Delta Lake 2.4.0 on <a href="https://spark.apache.org/releases/spark-release-3-4-0.html";>Apache Spark 3.4</a>. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.</p> <ul> <li>Documentation: <a href="https://docs.delta.io/2.4.0/";>https://docs.delta.io/2.4.0/</a></li> <li>Maven artifacts: <a href="https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.4.0/";>delta-core_2.12</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-core_2.13/2.4.0/";>delta-core_2.13</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.12/2.4.0/";>delta-contribs_2.12</a> <a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/2.4.0/";>delta_contribs_2.13</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-storage/2.4.0/";>delta-storage</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/2.4.0/";>delta-storage-s3-dynamodb</a></li> <li>Python artifacts: <a href="https://pypi.org/project/delta-spark/2.4.0/";>https://pypi.org/project/delta-spark/2.4.0/</a></li> </ul> <h3>The key features in this release are as follows</h3> <ul> <li>Support for <a href="https://spark.apache.org/releases/spark-release-3-4-0.html";>Apache Spark 3.4</a>.</li> <li><a href="https://redirect.github.com/delta-io/delta/issues/1591";>Support</a> writing <a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors";>Deletion Vectors</a> for the <code>DELETE</code> command. Previously, when deleting rows from a Delta table, any file with at least one matching row would be rewritten. With Deletion Vectors these expensive rewrites can be avoided. See <a href="https://docs.delta.io/2.4.0/delta-deletion-vectors.html";>What are deletion vectors?</a> for more details.</li> <li>Support for all write operations on tables with Deletion Vectors enabled.</li> <li><a href="https://github.com/delta-io/delta/commit/9fac2e6af313b28bf9cd3961aa5dec8ea27a2e7b";>Support</a> <code>PURGE</code> to remove Deletion Vectors from the current version of a Delta table by rewriting any data files with deletion vectors. See the <a href="https://docs.delta.io/2.4.0/delta-deletion-vectors.html#apply-changes-with-reorg-table";>documentation</a> for more details.</li> <li><a href="https://redirect.github.com/delta-io/delta/issues/1701";>Support</a> reading Change Data Feed for tables with Deletion Vectors enabled.</li> <li><a href="https://github.com/delta-io/delta/commit/7c352e9a0bf4b348a60ca040f9179171d2db5f0d";>Support</a> <code>REPLACE WHERE</code> expressions in SQL to selectively overwrite data. Previously “replaceWhere” options were only supported in the DataFrameWriter APIs.</li> <li><a href="https://github.com/delta-io/delta/commit/c53e95c71f25e62871a3def8771be9eb5ca27a2e";>Support</a> <code>WHEN NOT MATCHED BY SOURCE</code> clauses in SQL for the Merge command.</li> <li><a href="https://github.com/delta-io/delta/commit/422a670bc6b232e451db83537dcad34a5de97b67";>Support</a> omitting generated columns from the column list for SQL <code>INSERT INTO</code> queries. Delta will automatically generate the values for any unspecified generated columns.</li> <li><a href="https://redirect.github.com/delta-io/delta/pull/1626";>Support</a> the <code>TimestampNTZ</code> data type added in Spark 3.3. Using <code>TimestampNTZ</code> requires a Delta protocol upgrade; see the <a href="https://docs.delta.io/2.4.0/versioning.html";>documentation</a> for more information.</li> <li>Other notable changes <ul> <li>Increased resiliency for S3 multi-cluster reads and writes. <ul> <li><a href="https://redirect.github.com/delta-io/delta/pull/1711";>Use</a> a per-JVM lock to minimize the number of concurrent recovery attempts. Concurrent recoveries may cause concurrent readers to see a <code>RemoteFileChangedException</code>.</li> <li><a href="https://redirect.github.com/delta-io/delta/pull/1712";>Catch</a> any <code>RemoteFileChangedException</code> in the reader and retry reading.</li> </ul> </li> <li><a href="https://github.com/delta-io/delta/commit/303d640a";>Allow</a> changing the column type of a <code>char</code> or <code>varchar</code> column to a compatible type in the <code>ALTER TABLE</code> command. The new behavior is the same as in Apache Spark and allows upcasting from <code>char</code> or <code>varchar</code> to <code>varchar</code> or <code>string</code>.</li> <li><a href="https://github.com/delta-io/delta/commit/579a3151db611c5049e5ca04a32fc6cccb77448b";>Block</a> using <code>overwriteSchema</code> with dynamic partition overwrite. This can corrupt the table as not all the data may be removed, and the schema of the newly written partitions may not match the schema of the unchanged partitions.</li> <li><a href="https://github.com/delta-io/delta/commit/83513484";>Return</a> an empty <code>DataFrame</code> for Change Data Feed reads when there are no commits within the timestamp range provided. Previously an error would be thrown.</li> <li><a href="https://github.com/delta-io/delta/commit/5ab678db";>Fix</a> a bug in Change Data Feed reads for records created during the ambiguous hour when daylight savings occurs.</li> <li><a href="https://github.com/delta-io/delta/commit/28148976";>Fix</a> a bug where querying an external Delta table at the root of an S3 bucket would throw an error.</li> <li><a href="https://github.com/delta-io/delta/commit/81c7a58e";>Remove</a> leaked internal Spark metadata from the Delta log to make any affected tables readable again.</li> </ul> </li> </ul> <p>Note: the Delta Lake 2.4.0 release does not include the <a href="https://docs.delta.io/2.4.0/delta-utility.html#convert-an-iceberg-table-to-a-delta-table";>Iceberg to Delta converter</a> because <code>iceberg-spark-runtime</code> does not support Spark 3.4 yet. The Iceberg to Delta converter is still supported when using Delta 2.3 with Spark 3.3.</p> <h3>Credits</h3> <p>Alkis Evlogimenos, Allison Portis, Andreas Chatzistergiou, Anton Okolnychyi, Bart Samwel, Bo Gao, Carl Fu, Chaoqin Li, Christos Stavrakakis, David Lewis, Desmond Cheong, Dhruv Shah, Eric Maynard, Fred Liu, Fredrik Klauss, Haejoon Lee, Hussein Nagree, Jackie Zhang, Jintian Liang, Johan Lasperas, Lars Kroll, Lukas Rupprecht, Matthew Powers, Ming DAI, Ming Dai, Naga Raju Bhanoori, Paddy Xu, Prakhar Jain, Rahul Shivu Mahadev, Rui Wang, Ryan Johnson, Sabir Akhadov, Satya Valluri, Scott Sandre, Shixiong Zhu, Tom van Bussel, Venki Korukanti, Vitalii Li, Wenchen Fan, Xi Liang, Yaohua Zhao, Yuming Wang</p> <h2>Delta Lake 2.3.0</h2> <p>We are excited to announce the release of <a href="https://github.com/delta-io/delta/releases/tag/v2.3.0";>Delta Lake 2.3.0</a> on <a href="https://spark.apache.org/releases/spark-release-3-3-2.html";>Apache Spark 3.3</a>. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.</p> <ul> <li>Documentation: <a href="https://docs.delta.io/2.3.0/";>https://docs.delta.io/2.3.0/</a></li> <li>Maven artifacts: <a href="https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.3.0/";>delta-core_2.12</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-core_2.13/2.3.0/";>delta-core_2.13</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.12/2.3.0/";>delta-contribs_2.12</a> <a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/2.3.0/";>delta_contribs_2.13</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-storage/2.3.0/";>delta-storage</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/2.3.0/";>delta-storage-s3-dynamodb</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-iceberg_2.12/2.3.0/";>delta-iceberg_2.12</a>, <a href="https://repo1.maven.org/maven2/io/delta/delta-iceberg_2.13/2.3.0/";>delta-iceberg_2.13</a></li> <li>Python artifacts: <a href="https://pypi.org/project/delta-spark/2.3.0/";>https://pypi.org/project/delta-spark/2.3.0/</a></li> </ul> <h3>The key features in this release are as follows</h3> <ul> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1463";>Zero-copy</a> convert to <a href="https://docs.delta.io/2.3.0/delta-utility.html#convert-an-iceberg-table-to-a-delta-table";>Delta from Iceberg tables</a></strong> using <code>CONVERT TO DELTA</code>. This generates a Delta table in the same location and does not rewrite any parquet files. See the <a href="https://docs.delta.io/2.3.0/delta-utility.html#convert-an-iceberg-table-to-a-delta-table";>documentation</a> for details.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1505";>Support</a> <a href="https://docs.delta.io/2.3.0/delta-utility.html#shallow-clone-a-delta-table";><code>SHALLOW CLONE</code></a> for Delta, Parquet, and <a href="https://redirect.github.com/delta-io/delta/pull/1522";>Iceberg</a> tables</strong> to clone a source table without copying the data files. <code>SHALLOW CLONE</code> creates a copy of the source table’s definition but refers to the source table’s data files.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1555";>Support</a> idempotent writes for DML operations.</strong> This feature adds idempotency to <code>INSERT</code>/<code>DELETE</code>/<code>UPDATE</code>/<code>MERGE</code> etc. operations <a href="https://docs.delta.io/2.3.0/delta-batch.html#idempotent-writes";>using SQL configurations</a> <code>spark.databricks.delta.write.txnAppId</code> and <code>spark.databricks.delta.write.txnVersion</code>.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1511";>Support</a> <a href="https://docs.delta.io/2.3.0/delta-update.html#modify-all-unmatched-rows-using-merge";>“when not matched by source” clauses</a> for the Merge command</strong> to update or delete rows in the chosen table that don’t have matches in the source table based on the merge condition. This clause is supported in the Python, Scala, and Java <code>DeltaTable</code> APIs. SQL Support will be added in Spark 3.4.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1584";>Support</a> <a href="https://spark.apache.org/docs/3.2.2/sql-ref-syntax-ddl-create-table-like.html";><code>CREATE TABLE LIKE</code></a></strong> to create empty Delta tables using the definition and metadata of an existing table or view.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1604";>Support</a> reading Change Data Feed (CDF) in SQL queries</strong> using the <a href="https://docs.delta.io/2.3.0/delta-change-data-feed.html#read-changes-in-batch-queries";><code>table_changes</code></a> table-valued function.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/pull/1509";>Unblock</a> Change Data Feed (CDF) batch reads on column mapping enabled tables</strong> when <code>DROP COLUMN</code> and <code>RENAME COLUMN</code> have been used. See the <a href="https://docs.delta.io/2.3.0/delta-change-data-feed.html#change-data-feed-limitations-for-tables-with-column-mapping-enabled";>documentation</a> for more details.</li> <li><strong><a href="https://redirect.github.com/delta-io/delta/issues/1191";>Improved</a> read and write performance on S3</strong> when writing from a single cluster. Efficient file listing decreases the metadata processing time when calculating a table snapshot. This is most impactful for tables with many commits. <a href="https://docs.delta.io/2.3.0/delta-storage.html#single-cluster-setup-default";>Set the Hadoop configuration</a> <code>delta.enableFastS3AListFrom</code> to <code>true</code> to enable it.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/074ce173bb070ed60f10e98d3ef7dcc04cc2a744";><code>074ce17</code></a> Setting version to 2.4.0</li> <li><a href="https://github.com/delta-io/delta/commit/306efc3ae8565b633bf54f6ab8c321c3c043ed05";><code>306efc3</code></a> Handle FileAlreadyExistsException in S3DynamoDBLogStore</li> <li><a href="https://github.com/delta-io/delta/commit/5ee2d1a14a5484b16912a36f86ebf7bb27440c18";><code>5ee2d1a</code></a> Setting version to 2.4.0rc1</li> <li><a href="https://github.com/delta-io/delta/commit/0369c9e54753a543809df8fabbd43adfa92bff10";><code>0369c9e</code></a> Upgrade antlr4 to 4.9.3</li> <li><a href="https://github.com/delta-io/delta/commit/a4ac3edd13ce6e5fecdad8fbc217e16ed73a991e";><code>a4ac3ed</code></a> Add REORG operation details in commit history</li> <li><a href="https://github.com/delta-io/delta/commit/33faae01ca4e74bc07969cfe8f8fc3e53c3db824";><code>33faae0</code></a> Unify predicate strings in CommitInfo to record the information in a consiste...</li> <li><a href="https://github.com/delta-io/delta/commit/601704db70884167cd4978af8f2a959a41ad9c61";><code>601704d</code></a> Fix incorrect row index filter in CDCReader</li> <li><a href="https://github.com/delta-io/delta/commit/8a7045074f006d936ce01bd85ac7c8302af5342f";><code>8a70450</code></a> Write the number of deleted rows for DELETEs with DV</li> <li><a href="https://github.com/delta-io/delta/commit/296e76bcfecf3980c5d592f921a43b498f4428cf";><code>296e76b</code></a> Fix a test in DeltaVacuumSuite to pass locally</li> <li><a href="https://github.com/delta-io/delta/commit/c1bfa72bc5d0f7c079c6ecd47647562db10d00f5";><code>c1bfa72</code></a> Support insert-into-by-name for generated columns</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v2.2.0...v2.4.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
