alamb commented on code in PR #740:
URL: https://github.com/apache/arrow-site/pull/740#discussion_r2599072404
##########
_posts/2025-12-07-parquet-late-materialization-deep-dive.md:
##########
@@ -0,0 +1,310 @@
+---
+layout: post
+title: "A Practical Dive Into Late Materialization in arrow-rs Parquet Reads"
+description: "How arrow-rs pipelines predicates and projections to minimize
work during Parquet scans"
+date: "2025-12-07 00:00:00"
+author: hhhizzz and alamb
+categories: [application]
+translations:
+ - language: 简体中文
+ post_id: 2025-12-07-parquet-late-materialization-deep-dive-zh
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+This article dives into the decisions and pitfalls of implementing Late
Materialization in the [Apache Parquet] reader from [`arrow-rs`] (the reader
powering [Apache DataFusion] among other projects). We'll see how a seemingly
humble file reader requires complex logic to evaluate predicates—effectively
becoming a **tiny query engine** in its own right.
+
+[Apache Parquet]: https://parquet.apache.org/
+[Apache DataFusion]: https://datafusion.apache.org/
+[`arrow-rs`]: https://github.com/apache/arrow-rs
+
+## 1. Why Late Materialization?
+
+Columnar reads are a constant battle between **I/O bandwidth** and **CPU
decode costs**. While skipping data is generally good, the act of skipping
itself carries a computational cost. The goal of the Parquet reader in
`arrow-rs` is **pipeline-style late materialization**: evaluate predicates
first, then access projected columns. For predicates that filter many rows,
materializing after evaluation minimizes reads and decode work.
+
+The approach closely mirrors the **LM-pipelined** strategy from
[Materialization Strategies in a Column-Oriented
DBMS](https://www.cs.umd.edu/~abadi/papers/abadiicde2007.pdf) by Abadi et al.:
interleaving predicates and data column access instead of reading all columns
at once and trying to **stitch them back together** into rows.
+
+<figure style="text-align: center;">
+ <img src="{{ site.baseurl }}/img/late-materialization/fig1.png"
alt="LM-pipelined late materialization pipeline" width="100%"
class="img-responsive">
+</figure>
+
+To evaluate a query like `SELECT B, C FROM table WHERE A > 10 AND B < 5` using
late materialization, the reader follows these steps:
+
+1. Read column `A` and evaluate `A > 10` to build a `RowSelection` (a sparse
mask) representing the initial set of surviving rows.
+2. Use that `RowSelection` to read surviving values of column `B` and
evaluate `B < 5` and update the `RowSelection` to make it even sparser.
+3. Use the refined `RowSelection` to read column `C` (a projection column),
decoding only the final surviving rows.
+
+The rest of this post zooms in on how the code makes this path work.
+
+---
+
+## 2. Late Materialization in the Rust Parquet Reader
+
+### 2.1 LM-pipelined
+
+"LM-pipelined" might sound like something from a textbook. In `arrow-rs`, it
simply refers to a pipeline that runs sequentially: "read predicate column →
generate row selection → read data column". This contrasts with a **parallel**
strategy, where all predicate columns are read simultaneously. While
parallelism can maximize multi-core CPU usage, the pipelined approach is often
superior in columnar storage because each filtering step drastically reduces
the amount of data subsequent steps need to read and parse.
+
+The code is structured into a few core roles:
+
+-
**[ReadPlan](https://github.com/apache/arrow-rs/blob/bab30ae3d61509aa8c73db33010844d440226af2/parquet/src/arrow/arrow_reader/read_plan.rs#L302)
/
[ReadPlanBuilder](https://github.com/apache/arrow-rs/blob/bab30ae3d61509aa8c73db33010844d440226af2/parquet/src/arrow/arrow_reader/read_plan.rs#L34)**:
Encodes "which columns to read and with what row subset" into a plan. It does
not pre-read all predicate columns. It reads one, tightens the selection, and
then moves on.
+-
**[RowSelection](https://github.com/apache/arrow-rs/blob/bab30ae3d61509aa8c73db33010844d440226af2/parquet/src/arrow/arrow_reader/selection.rs#L139)**:
Describes "skip/select N rows" using either [Run-length encoding] (RLE)
(called a [`RowSelector`]) or a bitmask. This is the core mechanism that
carries sparsity through the pipeline.
Review Comment:
The idea of making them with HTML5 (so that they can be hand tweaked) is a
really good idea 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
