sandugood opened a new issue, #4944:
URL: https://github.com/apache/datafusion-comet/issues/4944
### Describe the bug
While testing out Comet on a rather large pipeline, querying Iceberg tables,
the execution dropped with an error:
```
Py4JJavaError: An error occurred while calling o12640.createOrReplace.
: java.lang.NegativeArraySizeException: -954238687
at
org.apache.comet.shaded.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:46)
at
org.apache.comet.serde.operator.CometIcebergNativeScan$.serializePartitions(CometIcebergNativeScan.scala:1025)
at
org.apache.spark.sql.comet.CometIcebergNativeScanExec.serializedPartitionData$lzycompute(CometIcebergNativeScanExec.scala:111)
at
org.apache.spark.sql.comet.CometIcebergNativeScanExec.org$apache$spark$sql$comet$CometIcebergNativeScanExec$$serializedPartitionData(CometIcebergNativeScanExec.scala:86)
at
org.apache.spark.sql.comet.CometIcebergNativeScanExec.commonData(CometIcebergNativeScanExec.scala:114)
```
Note that during `Spark` session building we distribute `Python` environment
to the executors by:
`"spark.archives": "/path/to/venv3.10.12.tar.gz#venv"`
And build the env like this (interactively inside JupyterNotebook cell):
```
! conda create -y -n myenv python=3.10.12
! conda run -n myenv pip install -r ../requirements.txt
! conda run -n myenv pip install conda-pack
! conda run -n myenv conda-pack -o venv3.10.12.tar.gz
```
Also, when running with log level set to "WARN", getting such messages:
`WARN DAGScheduler: Broadcasting large task binary with size 79.0 MiB`
It seems that it might be related to how Comet distributes `Python`
environment, or bags whole plan into a byteArray (which might exceed threshold)
Additional info:
- Iceberg V2
- Positional (row-level) deletes.
- MOR tables involved in the pipeline
### Steps to reproduce
_No response_
### Expected behavior
Comet doesn't crash on this query.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
