egalpin commented on PR #13742: URL: https://github.com/apache/pinot/pull/13742#issuecomment-2480045679
> I want to see what is the final desired way to handle multiple requests per request id. Do you plan to use a single request id on broker side and let server split it on a per table basis? I think that works, and actually opens up opportunity to do first level merge on the server side before sending the response to broker. My concept is to have a singular request ID associated with all the server requests that are issued. Each request sent to servers would be sent in the same way that it is today, where each tables settings of configs would be respected. From the perspective of the server, it would just be a set of queries like any other, as if a query to each table had been made in quick succession. Then, using the shared request ID, the broker would be able to reduce all the server responses from each targeted table down to a singular result table. We could definitely use multiple request IDs and track them from the Broker side, I hadn't previously considered that. The current approach, when in its final form, would also instantiate a distinct BrokerRequest per table associated with a logical table. That would be done because certain table settings might cause query re-writes, like enforcing approximation for distinct queries based on table-level settings. We could definitely get more intelligent about this and merge into a single server request any queries that have the same query contents (i.e. same re-write or no re-writes). Here's a rough example of how that might work (changes shown between this PR branch and another dev branch I have): https://github.com/egalpin/pinot/compare/egalpin/refactor-broker-request...egalpin:pinot:egalpin/refactor-broker-request-logical-table-expansion?expand=1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
