walterddr opened a new issue, #8668: URL: https://github.com/apache/pinot/issues/8668
Background === One of the frequently asked features of Pinot is the ability to retrieve the query result in small batches. For example, - users may perform resource-intense computations on the result on a row-by-row base; and materializing the entire result set in memory is wasteful. - support for some other systems like sql-alchemy (in https://github.com/python-pinot-dbapi/pinot-dbapi/) requires an engine to implement fetch-one and fetch-many Challenge === currently, Pinot executes these subset-of-result executions via the 2 argument limit clause; for example ``` SELECT * FROM myTbl limit 200,100 ``` returns 100 rows of the SQL result, starting from the 200th row. This poses a problem 1. it doesn't guarantee that the results are stable as the underlying data could've changed / or early termination of the limit kicks in and we have no idea which server returns the result set first. 2. it does the entire query execution again for each subset, e.g. `limit 0, 100` then `limit 100, 100` doesn't retain the previous execution result. Proposal === Support a simple cursor operation by enabling data caching in Pinot broker before serving back to the users. 1. Return results with the cursor-support turned on should associate the result with the requestID being executed. 2. Subsequence cursor operation can reuse the same requestID to seek for the next subset. 3. Pinot broker caches the cursor result set until it times out. Alternative Solution(s)? === Please comment below for any additional alternative solution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
