andygrove opened a new issue, #6176:
URL: https://github.com/apache/datafusion-comet/issues/6176
The library path passed to `CometNativeUDF.register` must already be valid
on every executor.
Comet does not ship the file anywhere; users have to bake it into their
image, mount it, or stage
it with their own tooling, and a path that exists only on the driver fails
at execution time.
Spark already has a mechanism for this: `SparkContext.addFile` /
`spark.files`, with
`SparkFiles.get(name)` resolving the local copy on each executor. `register`
could accept a library
added that way (or add it itself) and send the file name rather than an
absolute path, with the
planner resolving it through `SparkFiles` on the executor side.
Points to settle:
- Cache keying. The loaded-library cache is keyed by path and never unloads.
Two versions of a
library added under the same name must not collide, so the key probably
needs a content hash.
- Loading from `SparkFiles` directories, which differ per application,
interacts with the "never
copy over a loaded library" rule in the user guide.
Follow-up to #4459.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
