alamb commented on issue #24111: URL: https://github.com/apache/datafusion/issues/24111#issuecomment-5206277128
> Together these give users manual control first (enable dictionary reads, set a cardinality threshold), then build the statistics and optimizer support needed for DataFusion to make that same decision automatically. The end goal is a physical plan optimization that casts low-cardinality string columns to dictionary arrays before aggregation when the stats justify it, without requiring any manual intervention. I think one challenge with this approach is that casting to dictionary will likely be just as expensive as the grouping itself so I am not sure if this will actually save much work. we could potentially do something like add some heuristic of when to read from parquet as DictionaryArrays (as reading directly into Dictionary from parquet can be fast when it is physically encoded that way) I think the first step of any such project woudl be to make sure that DataFusion can take full advantage of dictionary arrays when the user specifies them -- then we can move on to trying to automatically chose to use them more frequently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
