Rich-T-kid commented on PR #23128: URL: https://github.com/apache/datafusion/pull/23128#issuecomment-4937006410
> We could potentially have an optimized dictionary group values that somehow wraps another group values and does the dictionary translation 🤔 This is pretty much what https://github.com/apache/datafusion/pull/23187 does, [benchmarks](https://github.com/apache/datafusion/pull/23187#issuecomment-4932432620) show good results > The thing with Dictionaries is that the dictionary indices for hte same logical value can vary from batch to batch, so we can't just do the grouping based on the dictionary indices I think theres room to optimize for the low cardinality case similar to what #21765 benchmarks showcased. Ive tried a couple approaches but the results have not been worth the extra complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
