andygrove opened a new issue, #7384: URL: https://github.com/apache/datafusion-comet/issues/7384
### What is the problem the feature request solves? Comet Native Executors (#7288) run a query only if all of it can run natively. Running part of a query in Spark and part on the cluster needs a shuffle that both engines can read and write. Spark's block transfer protocol could be that shuffle, without a separate shuffle service. Unlike Spark's RPC control plane, which uses Java serialization, the block transfer protocol in `network-common` has a compact binary encoding and has been stable for many releases. Comet's native shuffle already writes Spark's data and index file layout. Before designing any hybrid scheduling, it's worth finding out whether a native implementation of the data plane works against real Spark processes. ### Describe the potential solution An experiment, not production code: 1. Write a Rust client and server for the block fetch messages Spark uses to read shuffle blocks: `OpenBlocks`, `FetchShuffleBlocks`, `StreamHandle`, `ChunkFetchRequest` and `ChunkFetchSuccess`, and stream requests if needed, framed as `network-common`'s `MessageEncoder` and `MessageDecoder` frame them. 2. Using the client, fetch the blocks of a shuffle that Comet's native shuffle wrote, from a Spark 3.5 executor and from the external shuffle service, and decode them with Comet's shuffle reader. 3. Using the server, serve shuffle files written in Spark's layout, and read them from a Spark task through a shuffle whose map outputs point at the server. Authentication and encryption (SASL, `spark.network.crypto`) and push-based shuffle are out of scope. The experiment can require them to be off. The result should say which messages are needed, how the protocol differs across Spark 3.4 to 4.x, and what registering native map outputs with `MapOutputTrackerMaster` would take. ### Additional context The design notes are in the contributor guide's Comet Native Executors page, under "Speaking Spark's protocols" (added in #7289). A native block fetch client could also move Comet on Spark's shuffle reads out of the JVM. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
