vvivekiyer commented on issue #10658: URL: https://github.com/apache/pinot/issues/10658#issuecomment-1520518081
I have a [draft implementation for SUM_MV here](https://github.com/apache/pinot/compare/master...vvivekiyer:pinot:multistage_mv). I can improve on it and add support for all other aggregations in a similar fashion. To support GROUP BY, I am thinking of writing an RelOptRule to modify the return type of the project in the leaf stage to be a primitive type instead of array. But before I go ahead with making the entire changes, let’s discuss how we’d like to see the evolution of MV columns. I see that there are two possible options (please let me know if there are more): 1. Provide backward compatibility support for MV columns in the V2 Engine (for all existing operations). Create a new datatype for ARRAY. This will allow us to provide standard SQL operations (like postgres) on ARRAY columns. We will optionally decide to phase out MV column functionality in favor of standard ARRAYs (because MV functionality can be achieved with a combination of ARRAYs/UNNEST and subqueries). As the end state, Pinot will have support for two datatypes: - MV (which behaves the same way as today. Additionally working for Joins/Subqueries and other complex SQL). - ARRAY (which behaves similar to Postgres). 2. Do not introduce an ARRAY datatype. But equate/evolve our support of MV columns to achieve all the support that ARRAYs have. I’m inclined with doing (1). Also, currently, we unnest MV columns when applying aggregation functions and group by on MV columns. Do we want to retain the behavior as such when executing all sorts of complex SQL (Joins, Subqueries, etc)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
