egalpin commented on PR #13742:
URL: https://github.com/apache/pinot/pull/13742#issuecomment-2480045679

   > I want to see what is the final desired way to handle multiple requests 
per request id. Do you plan to use a single request id on broker side and let 
server split it on a per table basis? I think that works, and actually opens up 
opportunity to do first level merge on the server side before sending the 
response to broker.
   
   My concept is to have a singular request ID associated with all the server 
requests that are issued. Each request sent to servers would be sent in the 
same way that it is today, where each tables settings of configs would be 
respected. From the perspective of the server, it would just be a set of 
queries like any other, as if a query to each table had been made in quick 
succession. Then, using the shared request ID, the broker would be able to 
reduce all the server responses from each targeted table down to a singular 
result table.  We could definitely use multiple request IDs and track them from 
the Broker side, I hadn't previously considered that. 
   
   The current approach, when in its final form, would also instantiate a 
distinct BrokerRequest per table associated with a logical table.  That would 
be done because certain table settings might cause query re-writes, like 
enforcing approximation for distinct queries based on table-level settings.  We 
could definitely get more intelligent about this and merge into a single server 
request any queries that have the same query contents (i.e. same re-write or no 
re-writes).  Here's a rough example of how that might work (changes shown 
between this PR branch and another dev branch I have):  
https://github.com/egalpin/pinot/compare/egalpin/refactor-broker-request...egalpin:pinot:egalpin/refactor-broker-request-logical-table-expansion?expand=1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

Reply via email to