andygrove commented on PR #5051: URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5123240386
Thanks for the report @sandugood, and sorry for the bad error message — it sent you chasing scan configs when the failure was actually in the new cache serializer. @peterxcli diagnosed it correctly in the review thread above. What happened: the Comet cache serializer decided it could accept columnar input based on the schema alone. That makes Spark strip the `ColumnarToRow` above the cached plan and hand the serializer whatever that plan produces. When the cached plan falls back to Spark (or is an external columnar source like Iceberg), those are `On/OffHeapColumnVector`, not Arrow vectors, so cache materialization threw. Your "pipeline has lots of fallbacks to default Spark operators" was the trigger — and neither `spark.comet.scan.allowIncompatible` nor `spark.comet.convert.parquet.enabled` could help, because the failure is at cache write time, not in the scan. **Workaround on current main:** `spark.comet.exec.inMemoryCache.enabled=false` (the default), which keeps Spark's own cache serializer. **Fixed** in 9dce0d66 + c7bc49e5 on this branch: non-Arrow columnar batches are now copied into Arrow at cache write time, while Comet-native cached plans keep the existing zero-copy path. Two regression tests cover it — primitives/string/decimal/date/timestamp, and nullable array/struct/map/binary. Since Iceberg MOR is the one input I could not reproduce locally, it would be really helpful if you could retest your pipeline against the branch. Iceberg's vectors go through the same generic `ColumnVector` accessors as Spark's, so it should be covered, but confirmation from a real workload would be worth a lot here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
