sunchao opened a new issue, #5532:
URL: https://github.com/apache/datafusion-comet/issues/5532
### Describe the bug
Comet can fail while decoding malformed Base64 on rows that Spark never
evaluates. A limit may stop after the first row, or a semi/anti join may stop
checking candidates after finding a match. Both native `unbase64` and the JVM
codegen dispatcher can evaluate a whole batch before that early termination
takes effect.
This means enabling Comet can turn a successful Spark query into `Last unit
does not have enough valid bits`, even when the malformed value does not
contribute to the result.
### Steps to reproduce
Start Spark with the Comet extension, native execution, and the Comet
shuffle manager configured. Create the input with Comet disabled, then compare
the same query with Comet disabled and enabled:
```sql
SET spark.comet.enabled = false;
SET spark.sql.shuffle.partitions = 1;
CREATE TABLE unbase64_limit_input USING parquet AS
SELECT /*+ COALESCE(1) */ bad
FROM VALUES ('YWJj'), ('A') AS v(bad);
SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;
SET spark.comet.enabled = true;
SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;
```
In the reproduced single-file scan order, `YWJj` is read before `A`. Spark
returns `616263` (the bytes for `abc`). Comet instead throws on the second
value. This example relies on that physical input order; it does not assume
that SQL without `ORDER BY` guarantees ordering.
The failing Comet plan is:
```text
CometCollectLimit 1
+- CometProject [hex(unbase64(bad))]
+- CometNativeScan parquet
```
The same problem occurs when the decoder is in a filter below `LIMIT`, in
semi/anti join residual conditions, or below an already ordered
`WindowGroupLimit`. Compound inputs and `to_binary(bad, 'base64')` can reach
the JVM dispatcher and still fail for the same reason.
### Expected behavior
Match Spark's evaluation behavior: preserve successful queries when Spark
skips the malformed value, while still raising an error when Spark actually
decodes it. Ordinary native decoding should remain available when early
termination cannot skip the value.
### Additional context
Reproduced on Apache Comet `78defce020ac3c4d2b32ff7f172f3018b7587a48` with
Spark 4.1.3, a fresh native/JVM build, and verified Comet physical operators.
The relevant implementation is unchanged at
`a223ba14e9f1445b5d315962ae9005f7ec45b187`.
Seven query shapes produced 28 Spark-versus-Comet mismatches across all four
combinations of ANSI mode and AQE. Sixteen comparisons using valid inputs or
deliberately consumed malformed values matched Spark. This is not specific to
ANSI mode, and switching to JVM codegen dispatch alone does not preserve
row-by-row evaluation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
