alamb opened a new issue, #24149: URL: https://github.com/apache/datafusion/issues/24149
### Is your feature request related to a problem or challenge? While reviewing performance for https://github.com/apache/datafusion/pull/21566 from @kosiew I was trying to look at the performance for running clickbench q24 (which takes less than 100ms) To reproduce ```shell cargo build --bin benchmark_runner --profile=profiling samply record target/profiling/benchmark_runner clickbench --query 24 --iterations=20 ``` Here are the timings on my machine ``` clickbench/Q24 iteration 0: 99.0 ms, 10 rows clickbench/Q24 iteration 1: 69.2 ms, 10 rows clickbench/Q24 iteration 2: 69.7 ms, 10 rows clickbench/Q24 iteration 3: 68.5 ms, 10 rows clickbench/Q24 iteration 4: 70.6 ms, 10 rows clickbench/Q24 iteration 5: 69.2 ms, 10 rows clickbench/Q24 iteration 6: 67.7 ms, 10 rows clickbench/Q24 iteration 7: 69.3 ms, 10 rows clickbench/Q24 iteration 8: 68.3 ms, 10 rows clickbench/Q24 iteration 9: 67.9 ms, 10 rows clickbench/Q24 iteration 10: 70.0 ms, 10 rows clickbench/Q24 iteration 11: 68.9 ms, 10 rows clickbench/Q24 iteration 12: 68.3 ms, 10 rows clickbench/Q24 iteration 13: 71.1 ms, 10 rows clickbench/Q24 iteration 14: 71.1 ms, 10 rows clickbench/Q24 iteration 15: 70.3 ms, 10 rows clickbench/Q24 iteration 16: 68.8 ms, 10 rows clickbench/Q24 iteration 17: 69.0 ms, 10 rows clickbench/Q24 iteration 18: 70.7 ms, 10 rows clickbench/Q24 iteration 19: 71.6 ms, 10 rows ``` My reading of the profile is that almost 10% of the time is spent `clone` ing ParquetMetadata <img width="2348" height="1335" alt="Image" src="https://github.com/user-attachments/assets/7ac2503f-971f-4f15-aea9-89a60a75f0f2"; /> I think it is this line here: https://github.com/apache/datafusion/blob/24483dbeed7811a21480f307efd7becd87ce3111/datafusion/datasource-parquet/src/opener/mod.rs#L1686-L1687 I think what is happening is that reader is trying to load the page index. It sees there is no page index, so it then clones the metadata to try and load it. However, the clickbench file *has* no page index and so this always fails ### Describe the solution you'd like The code shouldn't be cloning the ParquetMetadata Ideally we would skip trying to load the page index if the file doesn't have any ### Describe alternatives you've considered I think ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
