adriangb commented on code in PR #21996:
URL: https://github.com/apache/datafusion/pull/21996#discussion_r3178163124
##########
datafusion/pruning/src/file_pruner.rs:
##########
@@ -70,16 +82,34 @@ impl FilePruner {
}
/// Create a new file pruner if statistics are available.
- /// Returns None if this file does not have statistics.
+ ///
+ /// Prefers the dense [`PartitionedFile::statistics`] when present, and
+ /// falls back to the sparse [`PartitionedFile::satisfied_stats`] map.
+ /// In the sparse case the accessors look up each
+ /// [`StatisticsRequest`] by key and lazily materialize the
+ /// single-row arrays the pruning predicate needs — no
+ /// densify-then-throw-away. Returns `None` if neither source has
+ /// any stats.
pub fn try_new(
predicate: Arc<dyn PhysicalExpr>,
file_schema: &SchemaRef,
partitioned_file: &PartitionedFile,
predicate_creation_errors: Count,
) -> Option<Self> {
- let file_stats = partitioned_file.statistics.as_ref()?;
- let file_stats_pruning =
- PrunableStatistics::new(vec![file_stats.clone()],
Arc::clone(file_schema));
+ let file_stats_pruning: Box<dyn PruningStatistics + Send + Sync> =
Review Comment:
`PruningPredicate` also uses this trait, so it might not be that hard to
port there too!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
