villebro commented on code in PR #2498:
URL:
https://github.com/apache/datafusion-ballista/pull/2498#discussion_r4125396205
##########
ballista/scheduler/src/state/session_manager.rs:
##########
@@ -84,3 +86,102 @@ pub fn create_datafusion_context(
Ok(Arc::new(SessionContext::new_with_state(session_state)))
}
+
+/// Wraps `session_builder` so that every session it builds shares one file
+/// statistics cache. [`BallistaCluster::new_memory`] applies this to the
+/// session builder it is given.
+///
+/// The scheduler builds a new session, with its own runtime, for every query.
+/// Planning a scan of a listing table collects statistics by reading the
+/// footer of every file in it, so with a cache per session every job pays for
+/// that again, which on large tables takes seconds. Cached statistics are
+/// checked against the size and modification time from each job's own file
+/// listing, so a file that has changed is read again. That check is why the
+/// listing cache must stay per session: sharing it too would serve stale
+/// statistics, and `COUNT(*)` is answered from them.
+///
+/// Sharing keeps statistics for the scheduler's lifetime instead of one job's.
+/// Entries are keyed by table and store-relative path, and the check compares
+/// neither e-tags nor versions. So a file rewritten in place at the same size,
+/// quickly enough that its modification time does not change, can still be
+/// served stale statistics, and so can a file in another store with the same
+/// path, size and modification time.
Review Comment:
Should we consider using etag or version in the case where those are
available, too? That would probably provide the best protection against
staleness. So compare in this order of preference:
1. if both sides have etag, just compare those (minimizes cache miss if the
content is byte identical)
2. if both sides have version, compare those (can cache miss if byte
identical change was pushed)
3. last resort: modification timestamp + size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
