Rich-T-kid commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5283980878
> Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option? my bad when I say RLE i'm referring to `RLE_DICTIONARY` > It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.). > I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users. I agree, ill update the PR to target all parquet scans. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
