danielwei816 opened a new issue, #7393:
URL: https://github.com/apache/datafusion-comet/issues/7393
### Describe the bug
Hey! We're running Comet 1.1.0 on EMR Serverless and hitting a
`NoSuchMethodError` when caching a query with Parquet inputs.
The failure is in `CometNativeScanExec.doCanonicalize()`. It calls
`FileSourceScanExec.copy()` with 10 arguments, but EMR's Spark build has a
12-argument version of that method.
Environment:
- EMR Serverless: `emr-spark-8.1.0`
- Spark: `4.1.1-amzn-0`
- Comet: `org.apache.datafusion:comet-spark-spark4.1_2.13:1.1.0`
- Java 17, ARM64
- Comet enabled, native shuffle and native Iceberg writes disabled
The error:
```text
java.lang.NoSuchMethodError:
'org.apache.spark.sql.execution.FileSourceScanExec
org.apache.spark.sql.execution.FileSourceScanExec.copy(org.apache.spark.sql.execution.datasources.HadoopFsRelation,
scala.Option, scala.collection.immutable.Seq,
org.apache.spark.sql.types.StructType, scala.collection.immutable.Seq,
scala.Option, scala.Option, scala.collection.immutable.Seq, scala.Option,
boolean)'
at
org.apache.spark.sql.comet.CometNativeScanExec.doCanonicalize(CometNativeScanExec.scala:298)
```
### Steps to reproduce
Our job joins Parquet inputs with Iceberg data, then calls:
```python
changes.persist(StorageLevel.MEMORY_AND_DISK)
```
Spark fails while preparing the cached query. I haven't reduced the full
query to a standalone PySpark reproducer yet, but the method mismatch can be
checked directly against the public EMR image:
```bash
docker run --rm --entrypoint /bin/bash \
public.ecr.aws/emr-serverless/spark4/emr-spark-8.1.0:20260827 \
-c 'javap -classpath "/usr/lib/spark/jars/*"
org.apache.spark.sql.execution.FileSourceScanExec | grep " copy("'
```
Compared with [upstream Spark
4.1.1](https://github.com/apache/spark/blob/v4.1.1/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L685-L695),
EMR adds two fields after `disableBucketedScan`:
- `enablePushdownAggregate: Boolean`
- `partitionOutput: Seq[Attribute]`
The EMR JAR has the 12-argument `copy()` and no 10-argument overload.
Comet's [call
here](https://github.com/apache/datafusion-comet/blob/1.1.0/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeScanExec.scala#L291-L300)
therefore can't resolve at runtime.
### Expected behavior
Caching the query succeeds, or Comet falls back to Spark for this scan when
the runtime is incompatible.
### Additional context
I realize EMR changes Spark internals and isn't part of Comet's tested Spark
matrix. I found #190 and the `makeCopy()` change in #207, which addressed a
similar issue on Databricks. This looks like the same assumption in a different
call site.
Would using `makeCopy()` with the runtime's existing field values be a
reasonable fix here? We haven't tested that patch yet.
_Drafted with
[Codex](https://openai.com/index/codex-now-generally-available/)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
