siddharthteotia commented on PR #15698: URL: https://github.com/apache/pinot/pull/15698#issuecomment-2848466737
Apologies if these questions have been discussed earlier. In that case, you can just point to the discussion or doc. I have a couple of questions out of curiosity -- - Are we primarily trying to optimize the JOIN ordering or broad set of queries ? - Have we looked at LOptJoinOptimizeRule in Calcite ? - I don't see a cost factory. We also don't have a stats store. So is this rule/HEP based ? > FWIW I don't expect this new optimizer to add any new overheads of its own. I think it really depends on the query complexity. In my experience, sophisticated query planning can easily go upto 100s of ms or even upto 1sec but then then it could be worth it if still optimizes the execution. I am very interested to know what are the kinds of queries for which you are seeing performance boost with the physical optimizer. > that is temporary: usePhysicalOptimizer=true/false. By default this is assumed false. When we use the physical optimizer, we need to skip certain sections of the HepProgram and I have made those changes Why is this needed? I am hoping that even in the first working version of the optimizer, there is no query option otherwise we are just going to create a lot of tech debt / conditional paths where people will inevitably bypass it and create their own niche optimizations in the OSS code. Overall I think this is a step in the right direction. It will make Pinot better and help make it feature complete. However, optimizer is a long journey (longer than execution) with lots of consistent non-trivial optimizations/revisions. I suggest we invest systematically with a long term direction which I am not sure what that is. If there is a doc, I would love to see it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
