geoffreyclaude commented on code in PR #26983:
URL: https://github.com/apache/datafusion/pull/26983#discussion_r4434695115
##########
datafusion/core/tests/fuzz_cases/sort_query_fuzz.rs:
##########
@@ -210,9 +211,10 @@ impl SortQueryFuzzer {
if expected_results.is_none() {
expected_results = Some(results);
} else if let Some(ref expected) = expected_results {
- // Every projected column is an ORDER BY key, so tied
- // rows have identical output and can be compared in
order.
- assert_eq!(expected, &results);
+ // Check logical equivalence.
+ // `assert_eq!` may report false failures when
logically equivalent
+ // batches have different physical encodings.
+ check_equality_of_batches(expected, &results).unwrap();
Review Comment:
Thanks @jayzhan211, agreed that we should preserve the ordering check.
Could we use Arrow’s `not_distinct` in `assert_same_rows_in_order` instead
of comparing formatted output? It compares corresponding values and correctly
handles the two dictionary null representations.
I tested this approach locally: the regression seed, determinism test and
randomized sort fuzzer all pass. Restoring direct batch equality reproduces the
original failure.
Keeping the existing batch concatenation, the helper could take individual
batches:
```diff
-use arrow::util::pretty::pretty_format_batches;
+use arrow::compute::kernels::cmp::not_distinct;
-/// Compares rows in order by their displayed values, so logically equal
-/// batches with different physical encodings (e.g. dictionaries) still
match.
+/// Compares values in row order, treating different physical
+/// encodings of the same logical types (e.g. dictionary nulls) as equal.
fn assert_same_rows_in_order(
- expected: &[RecordBatch],
- actual: &[RecordBatch],
+ expected: &RecordBatch,
+ actual: &RecordBatch,
msg: &str,
) {
- let expected = pretty_format_batches(expected).unwrap().to_string();
- let actual = pretty_format_batches(actual).unwrap().to_string();
- assert_eq!(expected, actual, "{msg}");
+ assert_eq!(expected.schema(), actual.schema(), "{msg}");
+ assert_eq!(expected.num_rows(), actual.num_rows(), "{msg}");
+
+ for (index, (left, right)) in
+ expected.columns().iter().zip(actual.columns()).enumerate()
+ {
+ let equal = not_distinct(left, right).unwrap();
+ assert_eq!(
+ equal.true_count(),
+ expected.num_rows(),
+ "{msg}: column {index}"
+ );
+ }
}
```
This would preserve the ordering check without making formatting part of the
correctness decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
