sunchao commented on PR #5854: URL: https://github.com/apache/datafusion-comet/pull/5854#issuecomment-5745846643
Reviewed `6962053`. **One P2 remains; I’d fix it before landing.** **[P2] Preserve policy changes outside whole-stage codegen** — [operators.scala:588](https://github.com/apache/datafusion-comet/blob/69620536200fb3b19284d0512aa83214818a7883/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala#L588-L589) The new lazy value always retains the first policy. Spark rebuilds its map builder in each task when the projection runs outside whole-stage codegen, including wide projections exceeding the default field limit. With AQE disabled and a 101-column projection: - Collect under `LAST_WIN`: both return the map. - Change to `EXCEPTION` and collect the same Dataset: Spark rejects the duplicate; Comet still returns the map. The reverse switch also diverges. Reproduced end to end for all three constructors in both directions, with wide projections and explicitly disabled whole-stage codegen. The previous `explain()` issue is fixed. All 36 existing map tests, six explain regression probes, six TopK comparisons, and 21 native tests passed. **CI:** only the labeling check exists for this head. GitHub reports merge conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
