dependabot[bot] opened a new pull request, #11911: URL: https://github.com/apache/iceberg/pull/11911
Bumps [io.delta:delta-spark_2.12](https://github.com/delta-io/delta) from 3.2.1 to 3.3.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-spark_2.12's releases</a>.</em></p> <blockquote> <h2>Delta Lake 3.3.0 (RC2)</h2> <p>We are excited to announce the release of Delta Lake 3.3.0 RC2! Instructions for how to use this release candidate are at the end of these notes. To give feedback on this release candidate, please post in the Delta Users Slack <a href="https://delta-users.slack.com/archives/CG9LR6LN4/p1714432283657109";>here</a> or create issues in our Delta <a href="https://github.com/delta-io/delta";>repository</a>.</p> <h2>Highlights</h2> <h3>Delta Spark</h3> <ul> <li><a href="https://redirect.github.com/delta-io/delta/issues/1959";>Support for Identity Column</a> to assign unique values for each record inserted into a table.</li> <li>Support VACUUM <a href="https://redirect.github.com/delta-io/delta/pull/3857";>LITE</a> to deliver faster VACUUM for periodically run VACUUM commands.</li> <li><a href="https://redirect.github.com/delta-io/delta/pull/3449";>Support for Row Tracking Backfill</a> to alter an existing table to enable Row Tracking. <a href="https://docs.delta.io/latest/delta-row-tracking.html";>Row Tracking</a> allows engines such as Spark to track row-level lineage in Delta Lake tables.</li> <li>Support for <a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#version-checksum-file";>enhanced table state validation</a> with version checksums and improved Snapshot initialization performance based on this checksum.</li> </ul> <h3>Delta UniForm</h3> <ul> <li>Support for enabling UniForm Iceberg on existing tables without rewriting the data files using <code>ALTER TABLE</code>.</li> </ul> <h3>Delta Kernel</h3> <ul> <li><a href="https://redirect.github.com/delta-io/delta/pull/3541";>Support</a> for reading Delta tables that have <a href="https://docs.delta.io/latest/delta-type-widening.html";>Type Widening</a> enabled.</li> </ul> <p>More detailed release notes on these exciting features as well as the other changes included in this release coming soon!</p> <h2>Artifacts</h2> <p>We have published the artifacts to a <a href="https://oss.sonatype.org/content/repositories/iodelta-1181";>staging repository</a>.</p> <ul> <li>Delta Spark artifacts: <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-spark_2.12/3.3.0/";>delta-spark_2.12</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-spark_2.13/3.3.0/";>delta-spark_2.13</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-contribs_2.12/3.3.0/";>delta-contribs_2.12</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-contribs_2.13/3.3.0/";>delta_contribs_2.13</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-storage/3.3.0/";>delta-storage</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-storage-s3-dynamodb/3.3.0/";>delta-storage-s3-dynamodb</a></li> <li>Delta UniForm artifacts: <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-iceberg_2.12/3.3.0/";>delta-iceberg_2.12</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-iceberg_2.13/3.3.0/";>delta-iceberg_2.13</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-hudi_2.12/3.3.0/";>delta-hudi_2.12</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-hudi_2.13/3.3.0/";>delta-hudi-2.13</a></li> <li>Delta Kernel artifacts: <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-kernel-api/3.3.0/";>delta-kernel-api</a>, <a href="https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-kernel-defaults/3.3.0/";>delta-kernel-defaults</a></li> </ul> <h2>How to use the Delta Spark Release Candidate</h2> <p>Download Spark 3.5 from <a href="https://spark.apache.org/downloads.html";>https://spark.apache.org/downloads.html</a>.</p> <p>For this release candidate, we have published the artifacts to a <a href="https://oss.sonatype.org/content/repositories/iodelta-1181";>staging repository</a>. Here’s how you can use them:</p> <h4>Spark Submit</h4> <ul> <li>Add <code>--repositories https://oss.sonatype.org/content/repositories/iodelta-1181</code> to the command line arguments.</li> <li>Example:</li> </ul> <pre><code>spark-submit --packages io.delta:delta-spark_2.12:3.3.0 --repositories https://oss.sonatype.org/content/repositories/iodelta-1181 examples/examples.py </code></pre> <p>Currently Spark shells (PySpark and Scala) do not accept the external repositories option. However, once the artifacts have been downloaded to the local cache, the shells can be run with Delta 3.3.0 by just providing the <code>--packages io.delta:delta-spark_2.12:3.3.0</code> argument.</p> <h4>Spark Shell</h4> <pre><code>bin/spark-shell --packages io.delta:delta-spark_2.12:3.3.0 \ --repositories https://oss.sonatype.org/content/repositories/iodelta-1181 \ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog </code></pre> <h4>Spark SQL</h4> <pre><code>bin/spark-sql --packages io.delta:delta-spark_2.12:3.3.0 \ --repositories https://oss.sonatype.org/content/repositories/iodelta-1181 \ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog </tr></table> </code></pre> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/f405c3fc4ea3a3ed420f58fb8581aa34e0f0826c";><code>f405c3f</code></a> Setting version to 3.3.0</li> <li><a href="https://github.com/delta-io/delta/commit/d5edfaba73c2c2b6b21024cbc294b07886f66797";><code>d5edfab</code></a> [DOC][3.3] Doc changes for InCommitTimestamps (<a href="https://redirect.github.com/delta-io/delta/issues/3979";>#3979</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/2e186b46749a592c8ea71d1b771daa21ba5f39a5";><code>2e186b4</code></a> [3.3][Docs] Update docs for 3.3 release (<a href="https://redirect.github.com/delta-io/delta/issues/3992";>#3992</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/9b15a8fc3abf5dc0a5ed2b27d665a91433a5a620";><code>9b15a8f</code></a> [Spark][3.3] Make Identity Column High Water Mark updates consistent (<a href="https://redirect.github.com/delta-io/delta/issues/3990";>#3990</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/fb3fd94c556c1b5e7eccc5680d5b5394a73cb9af";><code>fb3fd94</code></a> Update Delta uniform documentation to include ALTER enabling - 3.3 (<a href="https://redirect.github.com/delta-io/delta/issues/3963";>#3963</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/6177288dbfd815ebe2a4398215d0b2ff6c34819b";><code>6177288</code></a> [3.3][Kernel] Disable setting columnMapping/icebergCompatV2 for 3.3.0 (<a href="https://redirect.github.com/delta-io/delta/issues/3980";>#3980</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/2c210de86e94caafc90cf3e5be2011a89ac6f0dc";><code>2c210de</code></a> [3.3] Add Documentation for Vacuum LITE (<a href="https://redirect.github.com/delta-io/delta/issues/3987";>#3987</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/8c18520a52b962f5569ef09d4f023389dc77bf13";><code>8c18520</code></a> [Spark][3.3] Fix auto-conflict handling logic in Optimize to handle DVs (<a href="https://redirect.github.com/delta-io/delta/issues/3982";>#3982</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/9c967b9f638c32d18b11588dcdbb2dc827a64446";><code>9c967b9</code></a> [Kernel] [3.3] Remove Coordinated Commits from public API (<a href="https://redirect.github.com/delta-io/delta/issues/3938";>#3938</a>) (<a href="https://redirect.github.com/delta-io/delta/issues/3973";>#3973</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/f29aea0cbbb335236837c54a650df40dffbc6c6c";><code>f29aea0</code></a> [PROTOCOL][Version Checksum][3.3] Remove references to Java-specific Int.MaxV...</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v3.2.1...v3.3.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
