unikdahal opened a new issue, #6718:
URL: https://github.com/apache/datafusion-comet/issues/6718
### Describe the bug
A broadcast hash join on a string key returns fewer rows than Spark when the
build side is a `SELECT DISTINCT` over roughly 40,000 or more keys. Every build
key exists exactly once in the probe table, so the join count should equal the
number of build keys.
| build keys | Spark | Comet |
|---:|---:|---:|
| 1,000 | 1,000 | 1,000 |
| 10,000 | 10,000 | 10,000 |
| 40,000 | 40,000 | 39,996 |
| 100,000 | 100,000 | 99,988 |
| 500,000 | 500,000 | 499,940 |
| 1,000,000 | 1,000,000 | 999,880 |
| 3,000,000 | 3,000,000 | query fails |
About 0.012% of the keys are lost, growing with the build side. The same
keys without `DISTINCT` join correctly, as do shuffled hash joins
(`SHUFFLE_HASH`) and sort-merge joins (`MERGE`) with or without `DISTINCT`.
Integer and long keys built the same way are correct in all three join
strategies.
### Steps to reproduce
Probe table: 4,000,000 rows, key `id = concat('k', lpad(cast(n as string),
10, '0'))` for `n` in `[0, 4000000)`, written once with plain Spark as an
Iceberg table (`fz_str`).
```sql
SELECT /*+ BROADCAST(d) */ count(*) AS n, count(DISTINCT f.id) AS ids
FROM fz_str f
JOIN (
SELECT DISTINCT concat('k', lpad(cast(id as string), 10, '0')) AS id
FROM range(100000, 200000)
) d ON f.id = d.id
```
Expected `(100000, 100000)`; Comet returns `(99988, 99988)`.
Configuration: Spark 3.5.9, Iceberg 1.8.1, `local[4]`,
`spark.sql.adaptive.enabled=false`, `spark.sql.shuffle.partitions=4`, Comet
native shuffle, `spark.comet.scan.icebergNative.enabled=true`, 5g off-heap.
### Expected behavior
The same rows as Spark.
### Additional context
- Seen on `main` at `0ac4dadae7`.
- The build side without `DISTINCT` is correct, which points at the
`DISTINCT` aggregate's string output on its way into the broadcast rather than
the join itself; not yet confirmed.
- Not yet reduced to a Parquet-only case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
