walterddr commented on issue #10657:
URL: https://github.com/apache/pinot/issues/10657#issuecomment-1532100863
collecting some feedback from folks, please feel free to comment below as
well:
- Parser should be part of the external of planner so that
- it can be used in controller and broker to parse out tables for routing
- parser is also used by v1 engine so it make sense to single it out (and
it is invariant with the Planner)
- We should consider making the component modules so that it doesn't need to
force dependency on some other packages, for example:
- parser doens't need to depend on planner
- dispatcher doesn't need to depend on calcite.
- decision here is to make sure interfaces are clear, whether to group
them into separated modules can be refactor later if interfaces have clear
boundaries.
- `Planner` -> `LogicalPlanner`:
- there should be a conversion regarding such as RelTrait, or RelExchange
is being the only node looked at by dispatcher
- where does some planning info to
- *information that only affects the node->operator* then this should be
node-level info and make it used in PhysicalPlanner (on the server) (e.g. use
concurrent index map group-by or 2-way reducer/merger group-by, this decision
might not be able to be made on broker b/c it might need info like column
cardinality on segment level)
- *Information that not only affects the node but also the
upstream/downstream* then this decision should be made on the broker (e.g.
using sort-merge join requires sorting operation on both inputs; using hash
join requires materializing hash table on either left or right side)
- hint can always be attached, but the decision on where to use that hint
is decided with the rule above^
- PinotQueryPlanner and Plan should be a separate module
- the only goal achieved here is de-calcite --> convert RelNode into
format that doesn't depend on Calcite
- PinotQueryPlanner + LogicalPlanner can be replaced with a Cache later on
if we decide to run periodic tasks that doesn't have changes to the SQL itself
(e.g. speed-up planning)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
