GitHub user leborchuk added a comment to the discussion: [Proposal] Exploring vectorized execution for Cloudberry as a pure extension (vec_executor)
Thank you, all looks promising! Yes, I keep coming up with the idea of writing a vectorized engine, but I don't dare to start work here. Thank you for sharing the result, it's very interesting, and I think something really interesting can be done here. What kind of approach has developed in my mind. I may be wrong, because I haven't started experimenting or writing code yet. We have a seq scan that can return data in arrow format - iceberg and pax. Then, instead of immediately converting them to PG format, we leave them in arrow and use vector operations to perform further transformations. We do this until we encounter an unsupported operation or the second statement does not contain data in PG format. Visually, we sort of start performing vector-style operations from the very bottom of the execution plan and gradually move up, leaving the data in arrow format for as long as possible. volcano - style is not a problem here, because it's more about transferring control between nodes. Yes, we start with root and move recursively through branches to get results, but this does not prevent Clickhouse from building a very efficient engine. I really like velox. I would like to try this engine. Rust scares me a lot, to be honest. How I would like to use it is to learn how to perform individual execution plan operations using velox. First of all, these are hash join, group by, and data transfer in arrow format between slices. Replacing not individual slices, but individual operations. Because I want more control over the engine and I want to be able to add my own operations or improve the implementation of the engine to meet the requirements of Cloudberry. GitHub link: https://github.com/apache/cloudberry/discussions/2267#discussioncomment-19479071 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
