comphead opened a new issue, #3383: URL: https://github.com/apache/iceberg-rust/issues/3383
### Is your feature request related to a problem or challenge? Two steps in the Parquet read path do more work than needed on every `FileScanTask`. Both cost time that grows with the number of row groups or mapped fields. **1. Row-group filtering runs on every row group and then intersects two lists.** When row-group filtering is enabled and a task has a byte range, `ArrowReader::get_selected_row_group_indices` evaluates `RowGroupMetricsEvaluator` on every row group in the file. The result is then intersected with the byte-range selection using `Vec::contains` in `arrow/reader/pipeline.rs`. - The predicate is evaluated on row groups the byte range has already excluded. When a file is split into N tasks, each task evaluates the predicate over the whole file. - The `contains` intersection is `O(n * m)`. The bloom filter step in the same function already avoids this by taking the current selection as `candidate_row_groups` and evaluating only those. **2. Fallback name mapping allocates and scans linearly per Arrow field.** `apply_name_mapping_to_arrow_schema` in `arrow/reader/projection.rs` calls `names().contains(&field.name().to_string())` against every mapped field, for every Arrow field in a file without embedded field IDs. That allocates a `String` per comparison and costs `O(arrow fields * mapped fields)`. Java answers the same lookup from an index built once (`NameMapping.find` through `lazyFieldsByName`). Related: #2172 (ArrowReader performance for split `FileScanTask`s). ### Describe the solution you'd like 1. Pass the byte-range selection to `get_selected_row_group_indices` as `candidate_row_groups` and evaluate only those row groups. Each step narrows the previous step's output, so ordering is preserved without a separate intersection. 2. Build a `HashMap<&str, Option<i32>>` from the name mapping once per call and look Arrow fields up in it. Use `entry(...).or_insert(...)` so the first mapped field wins for duplicate names, as today. Behavior must not change. Add tests for a byte range combined with a predicate, and for duplicate aliases in a name mapping. The author of #3269 reported microbenchmark numbers in that PR's description (Apple M1 Pro, Criterion). For example, intersecting two equal lists of 8,192 row groups went from 3.17 ms to 12.79 µs, and name mapping with 2,048 fields went from 61.87 ms to 8.51 ms. The harnesses were removed from the PR, so these numbers are not reproducible from the repository. Real files have far fewer row groups, so end-to-end gains are likely smaller. That is an expectation, not a measurement. ### Willingness to contribute I can contribute to this feature independently. Fixed by #3269. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
