parthchandra opened a new issue, #7425: URL: https://github.com/apache/datafusion-comet/issues/7425
### What is the problem the feature request solves? Cached DataFrames are common in batch ETL. Comet's Arrow cache (`spark.comet.exec.inMemoryCache.enabled`) is experimental and off by default. With it off, a cached relation is read through Spark's `InMemoryTableScanExec`, and the operators above it can't run natively. ### Describe the potential solution Close the remaining correctness and stability items, then turn the cache on by default: - the in-memory cache audit epic (#6834) - the Spark 3.4 executor abort with the cache enabled (#7305) - memory accounting and spill behavior for large cached relations - a performance check against Spark's cache on wide and narrow schemas ### Additional context Because the serializer has to be installed at startup, the default has to be decided before the session starts, not per query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
