gortiz commented on issue #13760: URL: https://github.com/apache/pinot/issues/13760#issuecomment-2370637551
First of all, this is a great initiative. I've been working in Devo Inc, a company that is focused on the timeseries market (trying to compete with Splunk) and there we designed our own streaming SQL database that supports both SQL and LINQ expressions. The later is closer to the M3QL language proposed in the design document and the language most customers used to use. I've also created an internal proposal in StarTree to discuss about this feature. It can be read [here](https://docs.google.com/document/d/1YsxJHI58jtOoz9VtU90pUx-rKvS7ql3bpU07Kqylx9M). My document is not focused on the language itself. I believe SQL is expressive enough to be able to execute this kind of queries, but I understand that the market is not there and supporting M3QL and specially PromQL will be probably more useful to attract users. I have some concerns about the designed document: 1. My main concern about the proposed design document is that I don't think we need to add too many new concepts on top of the multi-stage query engine (MSQE). Instead, what I think we need to add is some key functions and some special operators (specifically for aggregate, window and join) that should work in streaming fashion. 3. I think the design is too broad and ambitious. I would split it in the following: 1. Add streaming capabilities to the current MSQE. Basically the tumble functions and window base aggregate, join and probably window functions. 2. Add the plugable syntax/protocol support. There is where we would think about how to add M3QL, etc. 3. Add the ability to extend aggregates. The current design assumes this is going to be needed to support other syntaxes, but I don't think this should be part of the time series design given that is also something that would help normal queries. We have recently added the ability to create new scalar and transformation functions using typesafe and it was a hard job to do, so I think we should think about long term support when adding a way to add aggregate functions. Anyway I know I'm late to the party and several key PRs have been merged so it doesn't make sense to discuss too much about the past, but please take me also in consideration for future developments. Again, I don't want to sound very pessimistic. I think this is an awesome feature for Apache Pinot and it is great to see it progress that fast! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
