dependabot[bot] opened a new pull request, #4259: URL: https://github.com/apache/iceberg-python/pull/4259
Bumps [ray](https://github.com/ray-project/ray) from 2.58.0 to 2.59.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/ray-project/ray/releases";>ray's releases</a>.</em></p> <blockquote> <h2>Ray-2.59.0</h2> <p><strong>Highlights</strong></p> <ul> <li><strong>Ray Data LLM & Ray Serve LLM are GA/Stable:</strong> the LLM APIs graduate to general availability this release (<a href="https://redirect.github.com/ray-project/ray/issues/65194";>#65194</a>), alongside an upgrade to vLLM 0.27.0 (<a href="https://redirect.github.com/ray-project/ray/issues/65351";>#65351</a>).</li> <li><strong>Ray Data external shuffle (shuffle_v2):</strong> the new shuffle backend gains an external, disk-backed mode — a new external shuffle runtime library and task/operator set (<a href="https://redirect.github.com/ray-project/ray/issues/64828";>#64828</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65144";>#65144</a>), planner integration (<a href="https://redirect.github.com/ray-project/ray/issues/65499";>#65499</a>), and Join & Aggregation support on external shuffle (<a href="https://redirect.github.com/ray-project/ray/issues/65897";>#65897</a>). The strategy was renamed from hash_shuffle_v2 to shuffle_v2 (<a href="https://redirect.github.com/ray-project/ray/issues/65411";>#65411</a>) and the compression setting from hash_shuffle_compression to shuffle_compression (<a href="https://redirect.github.com/ray-project/ray/issues/65590";>#65590</a>).</li> <li><strong>Token authentication:</strong> Ray now turns on token authentication by default for local clusters. ray.init() without an address enables authentication and generates a token at ~/.ray/auth_token when none exists. ray start --head enables authentication when a token is already available, and otherwise warns that it started an unauthenticated cluster. Remote and multi-node clusters are unchanged in this release. Set RAY_AUTH_MODE=disabled to opt out (<a href="https://redirect.github.com/ray-project/ray/issues/64755";>#64755</a>).</li> <li><strong>Security hardening:</strong> the dashboard redacts runtime_env values (which routinely carry credentials) in browser-facing endpoints (<a href="https://redirect.github.com/ray-project/ray/issues/65226";>#65226</a>), and read_hudi gained an unpickling guard preventing remote code execution (<a href="https://redirect.github.com/ray-project/ray/issues/65780";>#65780</a>).</li> <li><strong>Ray Core at scale:</strong> removed the GCS resource-view reset that caused placement-group retry storms on large clusters (<a href="https://redirect.github.com/ray-project/ray/issues/65271";>#65271</a>), hardened the worker-lease protocol (at most one CancelWorkerLease per pending lease, lease-id tombstones — <a href="https://redirect.github.com/ray-project/ray/issues/65613";>#65613</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65420";>#65420</a>), and batched local object frees (<a href="https://redirect.github.com/ray-project/ray/issues/65000";>#65000</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64037";>#64037</a>).</li> <li><strong>New Data sources:</strong> native ORC file reading (<a href="https://redirect.github.com/ray-project/ray/issues/64600";>#64600</a>), multi-path read_lance (<a href="https://redirect.github.com/ray-project/ray/issues/64560";>#64560</a>), and fps/resize parameters for read_videos (<a href="https://redirect.github.com/ray-project/ray/issues/64120";>#64120</a>).</li> </ul> <h1><strong>Ray Data</strong></h1> <h3><strong>🎉 New Features</strong></h3> <ul> <li>Add support for reading ORC files (<a href="https://redirect.github.com/ray-project/ray/issues/64600";>#64600</a>)</li> <li>Introduce external <a href="https://www.anyscale.com/blog/ray-data-shuffle-v2#disk-shuffle";>disk shuffle</a>: runtime library, tasks and operators, planner integration, and Join & Aggregations on external shuffle (<a href="https://redirect.github.com/ray-project/ray/issues/64828";>#64828</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65144";>#65144</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65499";>#65499</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65897";>#65897</a>)</li> <li>Support reading multiple paths in read_lance (<a href="https://redirect.github.com/ray-project/ray/issues/64560";>#64560</a>)</li> <li>Add fps and resize parameters to read_videos() (<a href="https://redirect.github.com/ray-project/ray/issues/64120";>#64120</a>)</li> <li>Add max_concurrent_calls_per_actor to ActorPoolStrategy (<a href="https://redirect.github.com/ray-project/ray/issues/65534";>#65534</a>)</li> <li>Introduce reporting for task USS distribution metrics (<a href="https://redirect.github.com/ray-project/ray/issues/65666";>#65666</a>)</li> </ul> <h3><strong>💫 Enhancements</strong></h3> <ul> <li>Iceberg: deduplicate read-task state, better decoded-size estimation, and bounded read-task memory (<a href="https://redirect.github.com/ray-project/ray/issues/64811";>#64811</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65042";>#65042</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64813";>#64813</a>)</li> <li>Count materialized data as downstream capacity, and return external consumer bytes for downstream-capacity backpressure (<a href="https://redirect.github.com/ray-project/ray/issues/66226";>#66226</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65474";>#65474</a>)</li> <li>Disable throttling for Union and Mix operators (<a href="https://redirect.github.com/ray-project/ray/issues/65242";>#65242</a>)</li> <li>Pin the fused map function for shuffle tasks (<a href="https://redirect.github.com/ray-project/ray/issues/65480";>#65480</a>)</li> <li>Chunk DataFrame emission in WebDatasetDatasource._read_stream and detect non-contiguous keys in read_webdataset (<a href="https://redirect.github.com/ray-project/ray/issues/65394";>#65394</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63407";>#63407</a>)</li> <li>Expose read-task resource args in read_webdataset (<a href="https://redirect.github.com/ray-project/ray/issues/65623";>#65623</a>)</li> <li>Auto-detect AzureFileSystem for Blob HTTPS URLs in download() (<a href="https://redirect.github.com/ray-project/ray/issues/63648";>#63648</a>)</li> <li>Groundwork for footer-based Parquet reads: footer types and online bin packer, FooterReader actor pool, and ListFiles pushdown state derived from the ReadFiles scanner (<a href="https://redirect.github.com/ray-project/ray/issues/65210";>#65210</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65273";>#65273</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65214";>#65214</a>)</li> <li>Groundwork for Data fault tolerance (1/n): add a linear application-layer lineage tracker (<a href="https://redirect.github.com/ray-project/ray/issues/65519";>#65519</a>)</li> </ul> <h3><strong>🔨 Fixes</strong></h3> <ul> <li>Add an unpickling guard to prevent RCE when reading Hudi (<a href="https://redirect.github.com/ray-project/ray/issues/65780";>#65780</a>)</li> <li>Fix Iceberg Dataset.count() returning wrong row counts (<a href="https://redirect.github.com/ray-project/ray/issues/65175";>#65175</a>)</li> <li>Fix LeRobot delta windows for multidimensional features (<a href="https://redirect.github.com/ray-project/ray/issues/65165";>#65165</a>)</li> <li>OpTask._cancel never passed force=True (<a href="https://redirect.github.com/ray-project/ray/issues/65389";>#65389</a>)</li> <li>Fix an incorrect timeout value in the autoscaler shutdown warning (<a href="https://redirect.github.com/ray-project/ray/issues/65317";>#65317</a>)</li> <li>Correct the physical-to-logical operator mapping type (<a href="https://redirect.github.com/ray-project/ray/issues/61790";>#61790</a>)</li> <li>Don't register out-of-scope callbacks on blocks passed through from input to output (<a href="https://redirect.github.com/ray-project/ray/issues/66369";>#66369</a>)</li> <li>Fix crash in preprocessors due to stale metadata in Arrow-backed Pandas dataframes (<a href="https://redirect.github.com/ray-project/ray/issues/66578";>#66578</a>)</li> </ul> <h3><strong>📖 Documentation</strong></h3> <ul> <li>Style and accuracy passes on the shuffle, join/save, and OOM-prevention pages (<a href="https://redirect.github.com/ray-project/ray/issues/65374";>#65374</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65372";>#65372</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65525";>#65525</a>, <a href="https://redirect.github.com/ray-project/ray/issues/65509";>#65509</a>)</li> <li>Clarify num_workers in the PyTorch docs and iter_torch_batches collate_fn usage (<a href="https://redirect.github.com/ray-project/ray/issues/65311";>#65311</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64535";>#64535</a>)</li> <li>Deprecations: ray_remote_args for read APIs (<a href="https://redirect.github.com/ray-project/ray/issues/65501";>#65501</a>) and for Dataset transformations (<a href="https://redirect.github.com/ray-project/ray/issues/65228";>#65228</a>); min_rows_per_file with partitioned Parquet writes (<a href="https://redirect.github.com/ray-project/ray/issues/63368";>#63368</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ray-project/ray/commit/dd83c3a703a5422d0b47f142cb380c6330817608";><code>dd83c3a</code></a> [cherry-pick][2.59.0][Data] Fix preprocessors crashing on Arrow-backed pandas...</li> <li><a href="https://github.com/ray-project/ray/commit/3a22cdbcf1b14055801596ae46ffcb29716e5335";><code>3a22cdb</code></a> [cherry-pick][2.59.0][Fix][Core/Auth] Pass auth token to image_uri containers...</li> <li><a href="https://github.com/ray-project/ray/commit/94236f29104e469beaa4172c0874edb45a279903";><code>94236f2</code></a> [cherry-pick][2.59.0][core][kuberay] Support configurable Kubernetes API auth...</li> <li><a href="https://github.com/ray-project/ray/commit/f9cc8c031231bcf90df1e1a6f17acb616e985264";><code>f9cc8c0</code></a> [cherry-pick][2.59.0][Data] Don't register out-of-scope callbacks on blocks p...</li> <li><a href="https://github.com/ray-project/ray/commit/633947e26ec8a1bc49013426fed32721ea7411e5";><code>633947e</code></a> [cherry-pick][2.59.0][Docs] Prevent dismissed announcement banners from flash...</li> <li><a href="https://github.com/ray-project/ray/commit/05d558dbe159a1ade631a7d882d0f4f61a258d60";><code>05d558d</code></a> [cherry-pick][2.59.0][Data] Count Materialized Data as Downstream Capacity (#...</li> <li><a href="https://github.com/ray-project/ray/commit/f27384545f717269fdabfb020e28c76c017bfe6e";><code>f273845</code></a> [cherry-pick][2.59.0][Feat][Core/Auth] Enable token authentication for local ...</li> <li><a href="https://github.com/ray-project/ray/commit/bfe9b7a5cdffa3d2341a694e7a606b7ac3e147d3";><code>bfe9b7a</code></a> [cherry-pick][2.59.0][ci][deps] Downgrade protobuf to 5.27.5 without the unre...</li> <li><a href="https://github.com/ray-project/ray/commit/e15d2899ee923e304dd76e175bceb9e862f2ff49";><code>e15d289</code></a> [cherry-pick][2.59.0][Java] Update httpcore5 to 5.4.3 (<a href="https://redirect.github.com/ray-project/ray/issues/66081";>#66081</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/66130";>#66130</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/150bc9bba98977e3ce63d44b6eff274b8d10bca2";><code>150bc9b</code></a> [cherry-pick][2.59.0][ci] Pin the uv installer version in release and CI imag...</li> <li>Additional commits viewable in <a href="https://github.com/ray-project/ray/compare/ray-2.58.0...ray-2.59.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
