Rich-T-kid opened a new issue, #26972: URL: https://github.com/apache/datafusion/issues/26972
### Is your feature request related to a problem or challenge? `enable_rle_to_dictionary` currently promotes a string or binary column when any row group has a dictionary page. Parquet writers can begin with dictionary encoding and fall back to plain data pages when the dictionary grows. DataFusion then exposes the column as a dictionary and rebuilds dictionaries for those plain pages during schema adaptation. This is particularly expensive for high-cardinality columns. In ClickBench Q23 (`SELECT * ... WHERE URL LIKE ... ORDER BY EventTime LIMIT 10`), a profile attributes 1,539 of 2,263 samples in `DecoderProjection::map` to `pack_byte_to_dictionary`, with 1,382 samples in `GenericByteDictionaryBuilder::append_array`. The `URL` column has plain fallback pages in 281 of 325 row groups. Directory schema merging compounds this: if one file reports a promoted dictionary field, the current normalization promotes compatible plain fields in every other file. ### Describe the solution you'd like Only infer a dictionary column when page encoding statistics show that every data page in every row group and every file is dictionary encoded. Missing page encoding statistics should fail closed and retain the plain value type. This keeps native dictionary reads for columns that can be decoded as dictionaries directly while avoiding generic per-batch dictionary construction for plain fallback pages. ### Additional context This is a narrow scan-side policy improvement related to #24117. A controlled ClickBench benchmark will compare the same DataFusion and Arrow revisions with this policy as the only code difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
