sunchao commented on code in PR #7089:
URL: https://github.com/apache/datafusion-comet/pull/7089#discussion_r4326631779
##########
native/core/src/parquet/eager_page_index_reader_factory.rs:
##########
@@ -504,15 +516,22 @@ fn mark_repeated_primitives(schema: &Schema, parquet:
&SchemaDescriptor) -> Sche
}
DataType::Map(entries, sorted) => DataType::Map(mark(entries,
columns, index), *sorted),
_ => {
- let repeated = columns.get(*index).is_some_and(|column| {
+ let column = columns.get(*index);
+ *index += 1;
+ let repeated = column.is_some_and(|column| {
column.self_type().get_basic_info().repetition()
== parquet::basic::Repetition::REPEATED
});
- *index += 1;
+ let nanos = column.is_some_and(is_nanos_timestamp);
Review Comment:
[P2] Preserve leaf ordinals through `ListView` and `LargeListView`. `mark`
does not recurse into either type, so a view containing multiple leaves
advances `index` only once. For a file with `a: ListView<Struct<x: int, n:
timestamp(ns)>>` followed by `t: INT96`, this assigns `a.n`'s NANOS marker to
`t`. Reading only `t` as `timestamp`, even without row-filter pushdown, now
fails with `Expected: timestamp, Found: INT64`. The supplied base and Spark
both return the timestamp successfully. Please recurse through both view
variants before applying the marker and cover this unrequested-column case with
a scan regression.
Evidence: Disposable test `review_int96_after_list_view` wrote a valid
Parquet schema with leaves `a.list.element.x INT32`, `a.list.element.n INT64
TIMESTAMP(NANOS,false)`, and `t INT96`. The `ARROW:schema` hint represented `a`
as ListView or LargeListView of the two-field struct. With `t` encoded as INT96
words `(1000000, 0, 2440589)`, requesting only `t: Timestamp(Microsecond, UTC)`
and disabling row filters failed for both variants at head. The identical test
passed at base `4a8a9496d119341b6f8ea94959312340c353583c`, returning
`1970-01-02T00:00:00.001+00:00`. Spark 3.5.9 returned the same value for both
files. Reproduction source and logs are
`/tmp/pr7089-41b859-review/listview_repro.rs`, `listview-head.log`,
`listview-base.log`, and `spark-listview.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
