neilconway opened a new issue, #25799:
URL: https://github.com/apache/datafusion/issues/25799
### Describe the bug
Dictionary-encoded values nested inside a struct or list change xxhash64's
result, so the same data hashes differently depending on its encoding. When the
first field of a struct, or any list element, is dictionary-encoded, the hash
accumulated so far is thrown away. Earlier arguments, and earlier list
elements, then have no effect on the result.
### To Reproduce
```
-- Dictionary-encoded field inside a struct: the first argument is dropped
SELECT xxhash64(7, named_struct('d', arrow_cast('a', 'Dictionary(Int32,
Utf8)'))) AS dict_7,
xxhash64(8, named_struct('d', arrow_cast('a', 'Dictionary(Int32,
Utf8)'))) AS dict_8,
xxhash64(7, named_struct('d', 'a')) AS plain_7,
xxhash64(8, named_struct('d', 'a')) AS plain_8;
-- Dictionary-encoded list elements: only the last element counts
SELECT xxhash64(make_array(arrow_cast('a', 'Dictionary(Int32, Utf8)'),
arrow_cast('b', 'Dictionary(Int32, Utf8)'))) AS
dict_list,
xxhash64(make_array('a', 'b')) AS plain_list,
xxhash64(make_array('b')) AS plain_last_only;
$ cargo run --bin datafusion-cli -- --spark -f repro.sql
+----------------------+----------------------+---------------------+---------------------+
| dict_7 | dict_8 | plain_7 |
plain_8 |
+----------------------+----------------------+---------------------+---------------------+
| -8582455328737087284 | -8582455328737087284 | 2701516948731497016 |
-694621655434011405 |
+----------------------+----------------------+---------------------+---------------------+
+----------------------+----------------------+----------------------+
| dict_list | plain_list | plain_last_only |
+----------------------+----------------------+----------------------+
| -6391946315847899181 | -7357372313250070212 | -6391946315847899181 |
+----------------------+----------------------+----------------------+
```
### Expected behavior
Dictionary encoding should not change the hash: dict_7/dict_8 should equal
plain_7/plain_8, and dict_list should equal plain_list.
### Additional context
Bug fixed in Comet already: apache/datafusion-comet#5757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
