hcrosse opened a new issue, #3618: URL: https://github.com/apache/iceberg-rust/issues/3618
### Apache Iceberg Rust version Main at `896f9d6c50ff52fa9e43aec170d3f14bfb40b512`, tested on Linux amd64 with Rust 1.99.1 and Arrow/Parquet 59.3.2. ### Describe the bug The default reader skips row groups that contain matching rows when a string column has only the deprecated `min`/`max` statistics that `parquet-mr` wrote before 1.10. Those bounds use signed byte order, so a value such as `é` sorts before `a`. I reproduced this with two synthetic rows, `a` and `é`, and deprecated bounds `min = é` and `max = a`. Filtering for `s = 'é'` returns no rows with the default reader and one row with row-group pruning disabled. The reader [converts the minimum directly to a string](https://github.com/apache/iceberg-rust/blob/896f9d6c50ff52fa9e43aec170d3f14bfb40b512/crates/iceberg/src/arrow/schema.rs#L1036-L1042) and [does the same for the maximum](https://github.com/apache/iceberg-rust/blob/896f9d6c50ff52fa9e43aec170d3f14bfb40b512/crates/iceberg/src/arrow/schema.rs#L1186-L1192). Neither conversion checks `is_min_max_deprecated`. The row-group evaluator then uses those values as string bounds. `parquet-java` uses deprecated bounds only when the type sorts as signed or `min` equals `max` ([`ParquetMetadataConverter`](https://github.com/apache/parquet-java/blob/3683bebbcfd51461e44c7212f4d9382513c06858/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L971-L990)). `parquet-rs` exposes these bounds with `is_min_max_deprecated()` set but does not drop them. `binary`, `fixed` and `uuid` bounds go through the same unchecked conversion, but I only tested `string`. For types that Parquet compares as unsigned bytes, such as `string`, [`get_parquet_stat_min_as_datum`](https://github.com/apache/iceberg-rust/blob/896f9d6c50ff52fa9e43aec170d3f14bfb40b512/crates/iceberg/src/arrow/schema.rs#L994-L1042) and [`get_parquet_stat_max_as_datum`](https://github.com/apache/iceberg-rust/blob/896f9d6c50ff52fa9e43aec170d3f14bfb40b512/crates/iceberg/src/arrow/schema.rs#L1144-L1192) could return no bound when the statistics are deprecated and `min` differs from `max`, as `parquet-java` does. ### To Reproduce Create a Rust 2024 binary crate with these dependencies in `Cargo.toml`. Put the program in `src/main.rs` and run `cargo run`. I used Arrow/Parquet 59.3.2, resolved from these version ranges. The program copies an encoded chunk into a new file with `SerializedRowGroupWriter::append_column`, replaces its statistics with deprecated-only bounds and sets `created_by` to `parquet-mr` 1.8.1. Older `parquet-mr` writers produce files with these signed-order statistics. Only the statistics and `created_by` imitate the old writer. The footer still declares a type-defined column order, and I did not test an original old-writer file. ```toml [dependencies] arrow-array = "59.2" arrow-schema = "59.2" bytes = "1" futures = "0.3" iceberg = { git = "https://github.com/apache/iceberg-rust";, rev = "896f9d6c50ff52fa9e43aec170d3f14bfb40b512" } parquet = "59.2" tokio = { version = "1", features = ["macros", "rt-multi-thread"] } ``` ```rust use std::collections::HashMap; use std::error::Error; use std::fs::File; use std::sync::Arc; use arrow_array::{RecordBatch, StringArray}; use arrow_schema::{DataType, Field, Schema as ArrowSchema}; use bytes::Bytes; use futures::TryStreamExt; use iceberg::Runtime; use iceberg::arrow::ArrowReaderBuilder; use iceberg::expr::{Bind, Predicate, Reference}; use iceberg::io::FileIO; use iceberg::scan::FileScanTask; use iceberg::spec::{DataFileFormat, Datum, NestedField, PrimitiveType, Schema, SchemaRef, Type}; use parquet::arrow::{ArrowWriter, PARQUET_FIELD_ID_META_KEY}; use parquet::column::writer::ColumnCloseResult; use parquet::data_type::ByteArray; use parquet::file::metadata::ParquetMetaDataReader; use parquet::file::properties::WriterProperties; use parquet::file::statistics::Statistics; use parquet::file::writer::SerializedFileWriter; #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { let path = std::env::temp_dir().join("deprecated_stats.parquet"); let path = path.to_str().unwrap(); write_file_with_deprecated_stats(path)?; let schema = Arc::new( Schema::builder() .with_fields(vec![ NestedField::required(1, "s", Type::Primitive(PrimitiveType::String)).into(), ]) .build()?, ); let s_is_e_acute = Reference::new("s").equal_to(Datum::string("é")); let reader = || ArrowReaderBuilder::new(FileIO::new_with_fs(), Runtime::current()); let full = rows(reader(), path, &schema, None).await?; println!("full scan: {full} row(s)"); for (label, reader) in [ ("default reader", reader()), ( "row-group filtering off", reader().with_row_group_filtering_enabled(false), ), ] { let n = rows(reader, path, &schema, Some(&s_is_e_acute)).await?; println!("s = 'é', {label}: {n} row(s), expected 1"); } Ok(()) } fn write_file_with_deprecated_stats(path: &str) -> Result<(), Box<dyn Error>> { let field = Field::new("s", DataType::Utf8, false).with_metadata(HashMap::from([( PARQUET_FIELD_ID_META_KEY.into(), "1".into(), )])); let file_schema = Arc::new(ArrowSchema::new(vec![field])); let batch = RecordBatch::try_new( file_schema.clone(), vec![Arc::new(StringArray::from(vec!["a", "é"]))], )?; let mut writer = ArrowWriter::try_new(Vec::new(), file_schema, None)?; writer.write(&batch)?; let source = Bytes::from(writer.into_inner()?); let metadata = ParquetMetaDataReader::new().parse_and_finish(&source)?; let column = metadata.row_group(0).column(0).clone(); // parquet-mr before 1.10 compared binary min/max as signed bytes, so "é" (0xC3 0xA9) sorts before "a". let stats = Statistics::new( Some(ByteArray::from("é")), Some(ByteArray::from("a")), None, Some(0), true, ); let props = WriterProperties::builder() .set_created_by( "parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)".into(), ) .build(); let mut writer = SerializedFileWriter::new( File::create(path)?, metadata.file_metadata().schema_descr().root_schema_ptr(), Arc::new(props), )?; let mut row_group = writer.next_row_group()?; row_group.append_column( &source, ColumnCloseResult { bytes_written: column.compressed_size() as u64, rows_written: 2, metadata: column.into_builder().set_statistics(stats).build()?, bloom_filter: None, column_index: None, offset_index: None, }, )?; row_group.close()?; writer.close()?; Ok(()) } async fn rows( reader: ArrowReaderBuilder, path: &str, schema: &SchemaRef, filter: Option<&Predicate>, ) -> Result<usize, Box<dyn Error>> { let size = std::fs::metadata(path)?.len(); let task = FileScanTask::builder() .with_file_size_in_bytes(size) .with_start(0) .with_length(size) .with_data_file_path(path.to_string()) .with_data_file_format(DataFileFormat::Parquet) .with_schema(schema.clone()) .with_project_field_ids(vec![1]) .with_predicate(filter.map(|p| p.bind(schema.clone(), true)).transpose()?) .with_case_sensitive(true) .build()?; let batches: Vec<RecordBatch> = reader .build() .read(Box::pin(futures::stream::iter([Ok(task)])))? .stream() .try_collect() .await?; Ok(batches.iter().map(RecordBatch::num_rows).sum()) } ``` Output: ```text $ cargo run # iceberg-rust 896f9d6c, parquet/arrow 59.3.2, rustc 1.99.1, linux/amd64 full scan: 2 row(s) s = 'é', default reader: 0 row(s), expected 1 s = 'é', row-group filtering off: 1 row(s), expected 1 ``` The fixture has no page index or bloom filter. The reproduction uses `FileScanTask` and local files, not catalog import or a complete `TableScan`. ### Expected behavior The default scan returns the matching row. Deprecated bounds on signed types such as `int` and `long` remain usable for pruning. ### Willingness to contribute I would be willing to contribute a fix for this bug with guidance from the Iceberg community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
