gruuya commented on code in PR #880: URL: https://github.com/apache/iceberg-rust/pull/880#discussion_r1908289944
########## crates/integrations/datafusion/src/statistics.rs: ########## @@ -0,0 +1,112 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::collections::HashMap; + +use datafusion::common::stats::Precision; +use datafusion::common::{ColumnStatistics, Statistics}; +use iceberg::spec::ManifestStatus; +use iceberg::table::Table; +use iceberg::Result; + +use crate::physical_plan::expr_to_predicate::datum_to_scalar_value; + +// Compute DataFusion table statistics for a given table/snapshot +pub async fn compute_statistics(table: &Table, snapshot_id: Option<i64>) -> Result<Statistics> { + let file_io = table.file_io(); + let metadata = table.metadata(); + let snapshot = table.snapshot(snapshot_id)?; + + let mut num_rows = 0; + let mut lower_bounds = HashMap::new(); + let mut upper_bounds = HashMap::new(); + let mut null_counts = HashMap::new(); + + let manifest_list = snapshot.load_manifest_list(file_io, metadata).await?; + + // For each existing/added manifest in the snapshot aggregate the row count, as well as null + // count and min/max values. + for manifest_file in manifest_list.entries() { + let manifest = manifest_file.load_manifest(file_io).await?; Review Comment: > Sorry, I don't get why it's not updated. If a row level deletion deletes a row with least values, should this be updated? Oh, so what I was getting at is that I'm not sure whether an existing data file's stats (upper/lower bounds) are updated when a row-level deletion occurs which targets that file. Ideally yes, but I guess that is not a given. In other words I'm wondering is there an equivalent to [Deltas tight-wide bounds](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#per-file-statistics) for Iceberg: > In the presence of [Deletion Vectors](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Deletion-Vectors) the statistics may be somewhat outdated, i.e. not reflecting deleted rows yet. The flag stats.tightBounds indicates whether we have tight bounds (i.e. the min/maxValue exists[1](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#user-content-fn-1-237f180ea194e75832a6e602cae5a094) in the valid state of the file) or wide bounds (i.e. the minValue is <= all valid values in the file, and the maxValue >= all valid values in the file). These upper/lower bounds are sufficient information for data skipping. Note, stats.tightBounds should be treated as true when it is not explicitly present in the statistics. ---------------------------- > For example, for vary length columns, the stored value maybe truncated Good point, manifests with trimmed upper/lower bounds for potentially large (e.g. string) columns will be misleading, arguably we should just skip providing statistics on those for now. > Also deletion not only change lower/upper bounds, but also row count, nvd, etc. True, though you can still get good estimates on those, and again it's more useful to provide those than not as hints to DataFusion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
