sunchao commented on PR #6170: URL: https://github.com/apache/datafusion-comet/pull/6170#issuecomment-5876853209
@andygrove Thanks, these are valid points. The four inline requests are addressed in 341174464, with replies in each thread. On the main scope/performance question: agreed that this does not enable native scans of collated-schema columns. The new kernel serves query-produced/cast collated values, and those input casts can still dispatch. I made that limitation explicit in the description. I ran the identical `arrayExtremaCollationBenchmark` fixture on #5403 head `ae755c459e0acd95f72a53f538d959e2adec572c` and this PR head `3411744640822ba46c36649b7171fe6df129fd80`, with release JNI libraries, Spark 4.1.3, JDK 17, local[1], and the same 262,144-row inputs. All 24 cases completed on each head, with assertions distinguishing dispatcher extrema on #5403 from native extrema here. The full Comet-versus-Comet mean-time table and methodology are now in the PR description. The comparison confirms an important tradeoff: Unicode LCASE/LCASE_RTRIM is about 1.9x faster, but short-ASCII collated cases mostly regress, long-ASCII LCASE is roughly level, and BINARY_RTRIM is slower on all three shapes (3.5-3.7x slower for long ASCII). Plain-binary controls improve 1.06-1.45x. So the previous Spark-versus-Comet table was not sufficient to justify a general speedup claim; the new results do not support one either. In particular, the RTRIM result remains a performance concern for the native-enablement decision, not something these test/proto changes fix. This is one local paired run and includes intervening upstream changes between the old head and rebased PR, not a strict single-patch ablation. Final-source validation passed 14 focused native checks and five Spark 4.1 JVM tests; broader CI is still pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
