comphead opened a new issue, #3472:
URL: https://github.com/apache/iceberg-rust/issues/3472
### Apache Iceberg Rust version
main (f83bec422)
### Describe the bug
A scan with an empty projection (`TableScanBuilder::select_empty()`, so
`FileScanTask::project_field_ids() == []`) reads every column of every data
file. An engine that only needs a row count, for example Spark's `df.isEmpty`
which becomes `select().limit(1)`, pays the full I/O cost of the table. Reading
only the Parquet footers would be enough.
Two places treat the empty list as "read everything".
`crates/iceberg/src/arrow/reader/pipeline.rs`: the metadata-only path
excludes the empty list.
```rust
let metadata_only_projection = !field_ids.is_empty()
&& field_ids
.iter()
.all(|id| PRUNABLE_METADATA_FIELDS.contains(id));
```
`crates/iceberg/src/arrow/reader/projection.rs`: the empty list falls
through to all columns.
```rust
if field_ids.is_empty() {
return Ok(ProjectionMask::all());
}
```
A comment in `pipeline.rs` says this is deliberate so that `COUNT(*)` keeps
its row count. The metadata-only path already gets the row count from the
`RowNumber` virtual column with `ProjectionMask::none`, so reading all columns
is not needed for that.
#566-era context: #676 / #677 added `select_empty()` but did not change the
reader.
### To Reproduce
Add this test to the `tests` module in
`crates/iceberg/src/arrow/reader/pipeline.rs`. It uses the existing
`write_parquet_with_wide_column`, `metadata_projection_task` and `scan_task`
helpers.
```rust
#[tokio::test]
async fn empty_projection_reads_no_data_columns() {
let tmp_dir = TempDir::new().unwrap();
let dir = tmp_dir.path().to_str().unwrap();
let empty = metadata_projection_task(
write_parquet_with_wide_column(dir, "empty.parquet", vec![], vec![]),
id_and_wide_schema(),
vec![],
);
let (batches, empty_bytes) = scan_task(empty).await;
let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
assert_eq!(rows, 3);
let wide = metadata_projection_task(
write_parquet_with_wide_column(dir, "wide.parquet", vec![], vec![]),
id_and_wide_schema(),
vec![2],
);
let (_, wide_bytes) = scan_task(wide).await;
assert!(
empty_bytes < wide_bytes,
"empty projection should read fewer bytes than the wide column:
{empty_bytes} vs {wide_bytes}"
);
}
```
Measured `bytes_read` on `main` for the same 3-row file:
| Projection | Bytes read |
|---|---|
| `[]` | 2838 |
| `[2]` (wide column only) | 2787 |
| `[_pos]` | 1928 |
The empty projection reads at least as much as the wide column alone, so the
test above fails.
### Expected behavior
An empty projection reads no data columns and still returns the correct row
count. With the `!field_ids.is_empty() &&` guard removed from
`metadata_only_projection`, the same file reads 1928 bytes, equal to the
`_pos`-only case, and still returns 3 rows with 0 columns.
### Willingness to contribute
I can contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
