jensholdgaard opened a new issue, #24769:
URL: https://github.com/apache/datafusion/issues/24769
> [!NOTE]
> This report was investigated and written with AI assistance (Claude Code),
posted with the account owner's review and consent.
## Describe the bug
DataFusion 54 prunes a Parquet row group from statistics alone when the
predicate is `col = <literal>` and the row group's statistics record
`null_count == row_count` for that column (equality with a non-NULL literal
cannot match any row). DataFusion 55.0.0 builds the **same pruning predicate**
— including the `<col>_null_count@N != row_count@M` clause — but no longer
prunes the row group; it is scanned instead.
Query **results are unaffected** (0 rows on both versions). The regression
is scan work: for workloads where a selective equality column is sparsely
populated (in our case, a `body` column that is NULL for the overwhelming
majority of rows), row groups that 54 skipped from the footer are now read.
## To Reproduce
Self-contained reproducer (~70 lines, inlined below): writes a
single-row-group Parquet file with one nullable `Binary` column, all 100 values
NULL, **default `WriterProperties` and default `SessionContext`**, registers it
as a `ListingTable`, and filters with the DataFrame API:
```rust
df.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?
```
Output, `datafusion = "54"`:
```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics:
PruningMetrics { pruned: 1, matched: 0, fully_matched: 0 } }
```
Output, `datafusion = "55"` (only the dependency line changed):
```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics:
PruningMetrics { pruned: 0, matched: 1, fully_matched: 0 } }
```
<details>
<summary>Cargo.toml + src/main.rs (complete)</summary>
```toml
[package]
name = "df-pruning-repro"
version = "0.0.0"
edition = "2021"
[dependencies]
# Flip to "54" and the row group is pruned; on "55" it is scanned.
datafusion = "55"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
tempfile = "3"
```
```rust
//! DataFusion 54 prunes a row group whose statistics say a column is
//! entirely NULL when the predicate is `col = <literal>`; DataFusion 55
//! scans it. Default writer properties, default SessionContext.
use std::sync::Arc;
use datafusion::arrow::array::{BinaryArray, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ScalarValue;
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::{
ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
};
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
// One row group, one nullable Binary column, every value NULL —
// the footer statistics record null_count == num_rows, no min/max.
let schema = Arc::new(Schema::new(vec![Field::new(
"body",
DataType::Binary,
true,
)]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(BinaryArray::from(vec![None::<&[u8]>; 100]))],
)
.unwrap();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("all_null.parquet");
let file = std::fs::File::create(&path).unwrap();
let mut w = ArrowWriter::try_new(file, schema, None).unwrap();
w.write(&batch).unwrap();
w.close().unwrap();
let ctx = SessionContext::new();
let url = ListingTableUrl::parse(format!("file://{}",
path.display())).unwrap();
let options =
ListingOptions::new(Arc::new(ParquetFormat::default())).with_file_extension(".parquet");
let schema = options.infer_schema(&ctx.state(), &url).await?;
let table = ListingTable::try_new(
ListingTableConfig::new(url)
.with_listing_options(options)
.with_schema(schema),
)?;
// `body = X` can match nothing when every value is NULL, so the row
// group is prunable from statistics alone.
let df = ctx
.read_table(Arc::new(table))?
.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?;
let plan = df.create_physical_plan().await?;
let batches = datafusion::physical_plan::collect(plan.clone(),
ctx.task_ctx()).await?;
println!(
"rows = {} (correct on both versions)",
batches.iter().map(|b| b.num_rows()).sum::<usize>()
);
fn walk(p: &Arc<dyn datafusion::physical_plan::ExecutionPlan>) {
if let Some(m) = p.metrics() {
for metric in m.iter() {
if metric.value().name() == "row_groups_pruned_statistics" {
println!("{:?}", metric.value());
}
}
}
for c in p.children() {
walk(c);
}
}
walk(&plan);
Ok(())
}
```
</details>
## Expected behavior
The row group is pruned on 55 as it was on 54: the file's statistics prove
`body = 'x'` cannot match (`null_count == row_count`), and the physical plan's
`pruning_predicate` (identical on both versions in our larger application,
including the `body_null_count != row_count` conjunct) already expresses that
proof.
## Additional context
- Found upgrading a Parquet log store (ourios) from DF 54.0.0 → 55.0.0:
three pruning-assertion tests went red with `pruned: 0` where 54 gave `pruned:
2`; written files byte-identical across the upgrade (parquet 58 vs 59 emit the
same statistics for these columns), so this is read-path only.
- Triage hint: the regression reproduces through **`ListingTable` + the
DataFrame API filter**. In our first attempt we could NOT reproduce via
`register_parquet` + a SQL string (`WHERE body = X'6E6F7065'`) — that path does
not prune on **either** 54 or 55 — so the SQL literal/rewrite path seems to sit
on a different guarantee/pruning route and may mask the regression during
triage.
- `EnabledStatistics::Page` vs `Chunk` and dictionary on/off for the column
make no difference; defaults reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
